Static

A false smoke alarm almost drowned Jane Street's only NYC datacenter

First reported by Michael.bacarella ·

The signal ●○○○ Compiled by AI from Michael.bacarella and Hacker News
Why you might care

Jane Street's critical infrastructure narrowly avoided destruction from a combination of environmental controls and safety systems.

What happened

A false smoke alarm in Jane Street's New York City datacenter nearly caused catastrophic damage. Contractors setting off the alarm inadvertently triggered a system that pressurized sprinkler pipes, which would have activated if heat sensors detected a fire. Crucially, the smoke alarm also shut down the datacenter's cooling systems (CRAC units). This created a dangerous scenario where rising temperatures in a uncooled room full of sensitive electronics could have led to the sprinklers activating and flooding the facility. A systems team member, the story's author, raced back to the office and manually pulled the power cords for every server rack to prevent overheating and a potential sprinkler deluge. The temperature stabilized at 143.6°F before sprinklers activated. The incident caused minimal downtime as it occurred after market close and systems were operational the next morning. Post-incident, Jane Street decided to retain its datacenter's in-office location, enhance procedures, and improve surveillance.

What it means

The incident highlights a critical vulnerability in the design of localized datacenters within office buildings, where environmental and fire suppression systems can interact in unexpected and destructive ways. Jane Street's decision to maintain servers adjacent to their trading floor, while initially driven by security and rapid shutdown capabilities, exposed them to a unique set of risks that standard datacenter practices might mitigate. The reliance on manual intervention for critical failures underscores a specific risk-management philosophy, which, while effective in this instance, suggests a gap in automated environmental fail-safes for such non-traditional datacenter locations.

The author's subsequent development of a custom OCaml monitoring tool to average server temperatures demonstrates a proactive response to identified system weaknesses. This custom solution addresses the limitations of existing telemetry and generic alerts, prioritizing a direct, reliable method for detecting widespread cooling failures. The choice of OCaml for its type safety and reliability suggests a growing trend towards using robust, specialized languages for critical infrastructure monitoring where even minor failures can have significant consequences.

AI-written summary. May contain errors.

Jane