More than 80% of Microsoft 365 network gateway servers became unresponsive during the global outage on November 12. Microsoft attributes the collapse to a defect introduced in July, which was triggered when traffic increased and nearly 14% of the systems’ capacity was out of service.
The disruption affected services such as Outlook, Exchange Online, Teams, and Copilot, and began at 11:20 a.m. U.S. Eastern time on Thursday, November 12. Microsoft confirmed full recovery at 11:10 a.m. the following day, after nearly 24 hours of problems that prevented access to email and delayed message delivery.
Three conditions converged on the gateways
Microsoft summarized the cause as three factors: a latent software defect, increased traffic on the gateways, and less available capacity than usual. The company explained that the combination of increased traffic and reduced capacity triggered the defect. This explanation appears in the preliminary report and in statements by Charles Lamanna, executive vice president of Copilot, Agents, and Platform.
The defect came from an update prepared for Microsoft 365 network gateway servers. The update was deployed starting in July and worked for more than three months without any problems being observed. However, under certain traffic conditions, it could cause lock contention and leave the machines unresponsive.
Microsoft explained that its usual load tests did not reproduce the combination that triggered the failure. The labs used traffic patterns and hardware configurations that differed from those in production; in addition, the tests focused on less powerful machines with fewer cores. The company said the problem was worse on machines with more cores and newer hardware.
“In less than two hours, the previously unknown defect was encountered across our fleet and rendered them unresponsive. At its lowest, more than 80% of servers were unresponsive.”
— Microsoft, preliminary report
The Teams routing change increased the load
About 15 hours before the outage, Teams automatically switched from an alternative gateway infrastructure to the primary one, which also supports Exchange Online. The failover changed the traffic pattern and increased the load on those servers. Microsoft noted that this type of switchover is part of its normal resilience operations.
Teams had also experienced a separate outage earlier that day, but Microsoft did not clarify whether the two incidents were related. What it did link to the subsequent outage was the routing change: by concentrating more traffic on the primary gateways, it left less headroom to absorb the effect of the latent defect.
Adding to the pressure, nearly 14% of gateway capacity was out of service because of an unrelated problem from the previous day. Microsoft maintained that the infrastructure had enough capacity to withstand that reduction while it was being resolved, and that those servers were expected to return to service through routine maintenance and health checks. The convergence of factors changed the outcome: more load reached fewer machines and triggered the failure.
Redirection turned the failure into a cascade
When some servers stopped responding, Microsoft’s systems redirected users to others that were still available. That response shifted requests, but it also overloaded the remaining servers. The same defect could then recur elsewhere in the fleet, and the outage spread in less than two hours.
flowchart TD
A["Defecto introducido en julio"] --> D["Más carga y menos capacidad"]
B["Teams cambia de gateway"] --> D
C["Casi 14% de capacidad fuera de servicio"] --> D
D --> E["Servidores dejan de responder"]
E --> F["Tráfico redirigido a otros servidores"]
F --> G["Más saturación en la flota"]
G --> H["Más del 80% sin respuesta"]Microsoft said that safeguards designed to prevent problems from spreading from one region to another also failed to contain this propagation. The episode exposed a specific tension in the recovery mechanism described by the company: diverting traffic can keep the service running if capacity is available, but in this case it shifted the load to servers susceptible to the same defect.
The Microsoft 365 architecture includes two separate and distinct gateway systems, designed for isolation and high availability. According to Lamanna, this design allows many services to switch between them during disruptions. However, Exchange Online could not switch to an alternative infrastructure, as other services—including Teams, OneDrive, and SharePoint—did, and those services were able to recover sooner.
Recovery also ran into accumulated load
Restarting the affected servers was not enough: Microsoft explained that they quickly received the accumulated traffic again and became overloaded once more. After identifying the defect, its engineers prepared a fix and deployed it to the affected servers. Services stabilized on Friday, and the company declared the active impact over at 11:10 a.m. Eastern time on November 13.
For businesses that rely on Exchange Online, the result was more than a brief degradation of an application: email became inaccessible and messages were delayed. The outage also affected Copilot and other Microsoft 365 services. The published information does not detail the impact on individual customers or quantify the delayed messages, so it does not allow comparisons of the impact across organizations.
More capacity and tests closer to production
Microsoft says that after the incident it increased available capacity, improved monitoring, and added safeguards to prevent traffic redirection from triggering similar cascades. The mismatch between test equipment and production servers also raises a clear operational question: in this case, the tests did not represent the combinations of hardware and traffic that exposed the defect.
The company’s explanation does not specify what specific changes it made to its load tests or how it will adjust redirection limits between servers or regions. It also does not determine whether the separate Teams outage was related in any way to the failover that increased traffic. Microsoft said it was continuing to investigate and expected to publish the final incident report later that week.
Sources: BleepingComputer, IT Pro