AI handles incidents, engineers lose touch with their systems
AI Signal Decode
The core issue highlighted is the "irony of automation," where advancements designed to simplify incident response paradoxically erode the practical skills of human engineers. As AI handles routine alerts, diagnoses, and even implements fixes, engineers receive fewer opportunities to develop the intuition and troubleshooting experience necessary for ambiguous or novel high-severity incidents. This creates a situation where automation's success makes humans less prepared for the rare but critical moments when it fails, echoing Lisanne Bainbridge's 1983 observations on human-factors in automated systems. The prediction is that while average MTTR might decrease, resolution times for complex, unforeseen issues will likely increase due to this skill degradation.
The article draws a parallel with the aviation industry, where pilots undergo extensive simulator training to prepare for rare, critical failures, despite advanced automation handling most flight operations. This is contrasted with the software industry, where such robust, scenario-based training for engineers is largely absent. Kalache advocates for the development and widespread adoption of incident simulation platforms, akin to those offered by Rootly and Uptime Labs. These simulations aim to replicate high-pressure incident scenarios, allowing engineers to practice crucial skills like decision-making with incomplete information, communication, coordination, and direct troubleshooting, thus bridging the gap left by reduced real-world incident exposure.
To counter the growing "comprehension debt" – the widening gap between system complexity and engineer understanding – Kalache stresses the need for engineers to actively engage with their systems. This involves more than just passive observation or explanations from AI. He posits that hands-on practice through incident simulations is essential for preserving and developing critical response skills, much like learning a sport requires active participation. Regularly facing unfamiliar failures, operating under pressure, and rehearsing coordination during simulated critical events are presented as vital components of on-call readiness in the age of increasingly capable AI incident responders.