Signal

Preliminary report: Microsoft blames a major Microsoft 365 and Exchange outage on a software defect introduced in a July 2026 update for network gateway servers

First reported by The Register ·

The signal ●●●○ Compiled by AI from The Register, Techmeme, 9to5Mac, Cyber Security News, PCMag and 2 more
Why you might care

Your Microsoft 365 services may be unreliable for days if they encounter unexpected load. Default Microsoft 365 testing procedures can miss critical defects that affect production. The fix for the outage took over fifteen hours to deploy and resolve.

What happened

Microsoft experienced a major outage affecting Microsoft 365 and Exchange Online services on November 12, 2026. A preliminary report attributes the widespread disruption to a software defect introduced in a July 2026 update for network gateway servers. This defect caused "lock contention" and exhausted server resources, particularly on newer, more powerful hardware. The outage was triggered when a Teams failover increased load on a primary gateway server simultaneously with approximately 14 percent of server capacity being offline. This combination pushed the affected servers into the defect, leading to a cascading failure where over 80 percent of servers became unresponsive within two hours. Standard recovery processes and rollbacks were ineffective due to the defect's age and the broad scope of the failure. Microsoft engineers developed an "update for the update" to revert the problematic code path, gradually restoring services over a period of approximately 15 hours. Exchange Online, which could not failover as easily as other services, experienced the longest downtime.

What it means

The incident highlights significant shortcomings in Microsoft's load testing methodologies, which focused on "weaker hardware with fewer cores." This implies that cloud providers may not be adequately simulating the conditions of their most powerful, modern infrastructure, leaving customers exposed to latent defects that only manifest under peak or unusual loads on high-end systems. The failure to implement a straightforward rollback mechanism for a months-old defect also points to challenges in managing the lifecycle of updates in vast, complex cloud environments.

The prolonged outage, particularly impacting Exchange Online due to its inability to gracefully failover, underscores the critical dependencies users have on the resilience and architectural design of cloud services. The fact that engineers had to create a "patch for the patch" rather than execute a simple rollback suggests that the underlying issue was deeply embedded or that the system's recovery mechanisms were not robust enough to handle such a systemic failure, indicating a potential area for future investment and development in cloud infrastructure resilience.

AI-written summary. May contain errors.