Microsoft on Monday said the global outage that knocked out Outlook, Teams and other Microsoft 365 services last week came down to three things going wrong at once — starting with a software bug that had been sitting in its systems, unnoticed, since July.
The company published a preliminary post-incident report on November 16, four days after the outage began. Charles Lamanna, Microsoft's executive vice president for Copilot, Agents and Platform, posted an abridged version on X and apologised to customers.
The first signs of trouble appeared around 1 p.m. Eastern time on Thursday, November 12. Within hours, Outlook mailboxes stopped responding, meeting invitations went missing and Teams features tied to email faltered. The disruption spread to users in more than 70 countries, according to Downdetector, which logged hundreds of thousands of complaints. Microsoft said service was restored to the remaining customers in the early hours of Saturday, November 14.
The report names three factors that, together, took down the networking gateway that sits in front of many Microsoft 365 services.
The first was the hidden defect. It came from a fix prepared specifically for the company's Microsoft 365 networking gateway servers. Deployment began in July, and the change ran worldwide for more than three months without any visible impact. The defect caused a lock contention problem inside an internal process, which left gateway servers overwhelmed and unresponsive under certain load conditions. Microsoft said its own load testing never caught it because the tests ran on weaker hardware than the production servers — and the defect hits harder on newer machines with more processing cores.
The second factor was extra load. About 15 hours before the outage, Microsoft Teams switched from an alternative networking gateway to the primary one, the same gateway that also serves Exchange Online, the company's email platform. That put more traffic on the shared gateway during peak hours in the United States.
The third was missing capacity. Nearly 14 percent of the gateway's server capacity was already out of service because of an unrelated issue from the day before.
With more traffic hitting fewer servers, the gateway ran into the latent bug from July. In under two hours, the failure had spread across the fleet. At its lowest point, more than 80 percent of the gateway servers were unresponsive, and the automated systems meant to contain the damage could not. Teams, OneDrive, SharePoint and Office apps recovered sooner. Exchange Online, which could not switch away from the gateway, took the longest.
Microsoft was clear about what the outage was not. The company said it was a technical issue, not a cyberattack, and not related to artificial intelligence. It said there was no evidence of malicious activity. During the outage, Microsoft extended the email timeout window to 48 hours to preserve delayed messages.
The scale explains why the outage drew so much attention. Microsoft 365 has more than 450 million commercial users, and the bundle sits at the centre of daily work for government offices, hospitals, police departments and companies around the world. Previous disruptions have rarely lasted more than several hours. This one ran about 36 hours from Microsoft's first status post to its all-clear.
A final post-incident review is due later this week, with more technical detail and the steps Microsoft plans to take to keep it from happening again. The preliminary report is available to administrators in the Microsoft 365 admin centre under incident references MO1550821 and EX1550543.



No comments:
Post a Comment