Signal

Microsoft blames monster email outage on software defect introduced in July fix

First reported by The Register ·

The signal ●●○○ Compiled by AI from The Register, Hacker News, The Verge, ZDNET, BleepingComputer and 12 more
Why you might care

You can no longer assume Microsoft's cloud services are resilient following a significant outage caused by a latent software defect.

What happened

Microsoft has identified a software defect introduced in a July update as the cause of a massive email and cloud service outage that affected users globally. The defect, which caused lock contention and exhausted server resources, was not detected during load testing because the tests focused on less powerful hardware. A surge in load due to a Teams failover, combined with approximately 14 percent of server capacity being out of service, triggered the defect in network gateway servers. This led to a cascading failure, rendering over 80 percent of servers unresponsive within two hours. Standard recovery and rollback procedures were ineffective, necessitating engineers to develop a patch for the faulty update. Service restoration began gradually after a mitigation was applied on November 13th, with full confirmation of healthy operations by November 14th.

What it means

The incident highlights critical flaws in Microsoft's testing and recovery processes, particularly the inadequacy of load testing on diverse hardware configurations and the failure of automated recovery systems to handle widespread unresponsiveness. This suggests a need for more robust validation before deploying updates to critical infrastructure, as even seemingly minor code changes can have catastrophic effects when combined with specific operational conditions. The reliance on a "patch for the patch" approach, rather than a swift rollback, further underscores the complexity and fragility that can arise in large-scale cloud environments.

This outage signals potential vulnerabilities across Microsoft's ecosystem, impacting not only Exchange Online but also other services like Teams, OneDrive, and SharePoint that rely on similar network gateway infrastructure. Companies utilizing Microsoft 365 services may need to re-evaluate their business continuity plans and consider the implications of prolonged service disruptions for their operations. The event also raises questions about the transparency and effectiveness of Microsoft's incident response communication and the underlying engineering practices that allowed such a defect to persist for months.

AI-written summary. May contain errors.