A July bug left Outlook and Teams down for nearly 24 hours

A July bug left Outlook and Teams down for nearly 24 hours

A software bug introduced in July left Outlook, Teams, and other Microsoft 365 services unavailable for 23 hours and 50 minutes, according to Microsoft's preliminary report. The outage began on November 12 and ended the following day; the company attributed the collapse to a combination of a latent bug, increased traffic at a network gateway, and reduced capacity.

The incident affected users around the world, including businesses, hospitals, and public agencies, and also disrupted Exchange Online and Copilot. In its preliminary report, Microsoft ruled out an intrusion as the cause: it describes a sequence of technical problems that eventually spread between servers and regions.

A July bug that testing failed to detect

The defect appeared in an update for Microsoft 365 network gateway servers. The update began rolling out in early July and had been in service for more than three months without any problems being observed. Under certain traffic conditions, however, an internal process would become resource-constrained and could cause machines to stop responding.

Charles Lamanna, Microsoft's executive vice president, explained that load tests did not reproduce the conditions that exposed the bug. The labs used a different mix of traffic and SKU, as well as less powerful hardware with fewer cores than production servers. The problem was actually worse on newer machines with more cores.

The operation that increased the load took place about 15 hours before the outage: Teams moved from alternative infrastructure to the main infrastructure shared with Exchange Online. The change was part of the infrastructure's normal operation, but it concentrated more traffic on a gateway that already had less capacity available.

Microsoft said that about 14% of the gateway's capacity was out of service because of an unrelated problem that had occurred the previous day. BleepingComputer and Engadget cite that figure from the preliminary report. Computer Hoy, however, says that more than 85% of the gateway servers were out of service because of that earlier incident; this figure does not match the 14% reduction in capacity cited by the other publications.

TimeReported event
Early JulyRollout begins for the update containing the latent bug.
About 15 hours before the outageTeams switches to the main gateway also used by Exchange Online.
November 12, 17:20The outage begins, according to SmartWorld.
November 13, 17:10The active impact ends, according to SmartWorld.
November 14Technical checks continue, according to SmartWorld.

BleepingComputer gives the start and end times as 11:20 a.m. ET on November 12 and 11:10 a.m. ET the following day. Microsoft said the outage coincided with the peak traffic period in the United States, on a Thursday afternoon.

Redirected traffic spread the failure

When the first servers stopped responding, traffic management systems sent requests to other available machines. But those machines received more load and encountered the same defect. The process turned a localized failure into a cascading failure: in less than two hours, more than 80% of the servers involved were unresponsive, according to Microsoft.

The mechanisms designed to contain problems between regions also failed to prevent the outage from spreading. Computer Hoy reports Lamanna's explanation of the propagation from server to server, while BleepingComputer notes that regional safeguards failed to isolate the incident. Several Microsoft 365 services around the world were affected.

Recovery could not be achieved simply by restarting the machines, either. Once they came back online, they received requests that had accumulated during the outage and could become blocked again. Because the defect had been in production for months, Microsoft could not simply remove it by rolling back a recent update: its engineers had to prepare a specific fix and distribute it gradually.

Teams, OneDrive, and SharePoint recovered before Exchange Online, according to BleepingComputer and SmartWorld. Exchange had less capacity to divert traffic to another infrastructure, which prolonged email access problems. Microsoft restored services by region, according to Computer Hoy. BleepingComputer reported that the company deployed a fix to the affected servers and that services stabilized during Friday. Microsoft confirmed that the active impact had ended on November 13 at 11:10 a.m. ET.

“We deeply regret the impact on all our customers and recognize the consequences for their operations and day-to-day activities.”

— Charles Lamanna, Microsoft's executive vice president

Phishing emails took advantage of the outage

While the outage was still ongoing, a high-volume phishing campaign began on November 13. Microsoft Threat Intelligence documented it on November 16, according to SmartWorld. The messages impersonated official Microsoft 365 status notices and claimed that some emails had been held up during the outage and would be deleted within 24 hours unless recipients recovered them.

The link supposedly used to restore those messages led to sites unrelated to Microsoft; the aim of stealing business account credentials was considered a possibility. SmartWorld said there was no confirmation that passwords had been stolen or accounts compromised, and that Microsoft Defender had detected and blocked messages and domains associated with the campaign.

Credits for businesses and calls for compensation

Microsoft is offering business customers credits ranging from 25% to 100% of the cost of the affected service, according to SmartWorld. They are not applied automatically: administrators must request them through the Microsoft 365 portal, under “Help and support,” before December 31, and provide the incident codes EX1550543 and MO1550821.

Individual Microsoft 365 Personal or Family subscribers are not eligible for automatic compensation. They can contact support to request it, but Microsoft decides whether to grant it. Codacons called for automatic compensation for businesses and citizens and threatened legal action; SmartWorld notes that this is an extraordinary claim, not an automatic refund right recognized in Microsoft's agreements. The company had not announced an extraordinary compensation plan.

A pool alert that never reached staff

In Brühl, near Cologne, an automatic system detected an excessive chlorine concentration in a hotel swimming pool and generated an alert for staff. The alert was sent by email and did not arrive during the outage. Fifteen guests suffered respiratory irritation; four were taken to the hospital and one was kept under observation, according to SmartWorld.

The incident highlighted the risk of relying on a single digital channel for safety communications. Microsoft's preliminary report already details the failure and its spread, but the company still needs to explain in its final report why the mechanisms for isolating regions did not contain the cascade. BleepingComputer reported that Microsoft had also increased capacity, improved monitoring, and added safeguards to prevent traffic redirection from causing a similar failure.

Sources: SmartWorld, Engadget, Computer Hoy, BleepingComputer