Incident: Microsoft 365 global degradation, November 12–14, 2026 · Advisories: EX1550543 (Exchange Online), MO1550821 (Microsoft 365 suite) · Verdict: a self-inflicted capacity death spiral executed by Microsoft's own automation. The health-check system was simultaneously the judge, the jury, and the executioner of fleet capacity — and it had no circuit breaker on its own eviction rate.
Microsoft's official post-incident report has not been published yet. What follows is a reconstruction from Microsoft's own status updates, independent network telemetry, and the documented blast radius — with inference labeled as inference. We will update this piece when the PIR lands.
The Incident at a Glance
| Field | Value |
|---|---|
| Start | 18:26 UTC, Thursday November 12, 2026 (advisory MO1550821). ThousandEyes observed connection resets, refusals, timeouts and packet loss inside Microsoft's network from roughly 17:30 UTC. |
| Duration | 27+ hours to broad recovery; most tenants experienced close to 24 hours of disruption. Residual degraded features and delayed email persisted into Saturday November 14. |
| Blast radius | 15+ hosted services — everything that touches Exchange Online: Outlook, Teams meetings, SharePoint Online, OneDrive, Microsoft 365 Copilot, cloud PCs, Intune, Defender for Office 365/XDR. Services that do not depend on Exchange Online stayed functional. |
| Demand signal | Downdetector logged 40,000+ problem reports from 70+ countries on Friday, with 43,000+ Outlook reports from users in 72 countries at the Friday-morning peak. |
| Stated root cause | Per Microsoft's Friday statement: a fault introduced during an earlier network capacity upgrade, surfaced after a more recent network change. Explicitly ruled out: cyberattack, traffic surge, AI/agents. |
| Status of analysis | Microsoft promised a full Post-Incident Report covering root cause and defense-in-depth actions. PIR pending at time of writing. |
Executive Summary
At 18:26 UTC on November 12, a portion of Microsoft's Exchange Online infrastructure began failing automated health checks and dropped out of the load-balancer rotation. Traffic did not increase — capacity left. The surviving servers absorbed the displaced load, a core process on each of them (the one that establishes secure connections) became overwhelmed, CPU saturated, probe deadlines were missed, and the health-check system evicted those servers too. This is a textbook positive feedback loop, executed by the very automation that exists to protect availability.
Recovery then failed repeatedly for the same reason: every server that was restarted, passed its probe, and rejoined the pool was immediately fed the queued storm of reconnecting clients — and died again. Microsoft's own status updates capture the loop: servers that returned to service "can become unresponsive again once they begin receiving traffic."
Three engineering failures compound into one global outage. First, a latent network fault from an earlier capacity upgrade was exposed by a more recent network change. Second, the health-check layer had no panic threshold, no eviction-rate cap, and no gray-failure concept — it removed capacity until none remained. Third, the TLS handshake path turned re-entry into a CPU wall: millions of clients reconnecting simultaneously forced full asymmetric handshakes on a fleet that was already drowning.
The Full Timeline, Reconstructed
Assembled from Microsoft's advisories as relayed by The Register, BleepingComputer, ABC, CNBC, CNN and Microsoft's X account. Times are UTC.
| Time (UTC) | Event |
|---|---|
| ~17:30, Nov 12 | ThousandEyes begins observing availability issues globally: connection resets, refusals, response timeouts, HTTP server errors, and significant traffic loss inside Microsoft's network. |
| 18:26, Nov 12 | Advisory MO1550821 opens. First diagnosis: "elevated load across multiple regions." The Americas are hit first; it is the start of the US working afternoon. |
| ~19:00–21:00 | Microsoft rules out a traffic surge: the elevated load "isn't driven by an increase in traffic, but by a number of servers being unexpectedly unavailable; straining the remaining infrastructure." Identifies automated health-check failures; suspects "a specific component that's responsible for processing server health." |
| ~21:00 | Microsoft says it has "validated an alternative service health check method" — some affected servers return to service. Automated recovery systems are being prepared to continually detect and restart unresponsive servers. |
| ~00:20, Nov 13 | Microsoft begins reverting "a recent network configuration change that correlates with the start of impact," monitoring whether the rollback helps. |
| ~03:15 | Rollback update: the configuration change being deployed is "not taking as expected." Microsoft confirms a "core process on affected servers is becoming overwhelmed, causing capacity loss," and starts isolating a portion of infrastructure, restarting affected components, and reintroducing traffic to validate stability. Admission: "our actions taken thus far have not provided the anticipated relief." |
| ~05:00 | Scaling up virtual machines to add capacity — but engineers hit "machines being stuck in a bad state, preventing our changes from syncing as expected." |
| 06:56 | Microsoft reports only "marginal success" after patching and restarting servers. |
| ~07:26 | A patch to an upstream authentication/routing configuration component shows "positive results for server stability" on a portion of infrastructure; being extended to servers in active regions as they come online. |
| ~07:00–09:00 | Downdetector peak: 43,000+ Outlook reports from 72 countries. Microsoft reports "early signs of stabilisation" at 09:24 but capacity has "not yet returned to normal levels." |
| 11:11 | Recovery work complete on parts of infrastructure; traffic gradually being restored. |
| 12:09–13:32 | Progress in Europe, then broader improvement in North America and early signs of recovery in Asia Pacific. Recovery is explicitly phased and sequential. |
| 14:17 | Email queues "significantly reduced and approaching normal levels." The 48-hour email retry timeout is protecting delayed messages; backlog clearing is prioritized. |
| Nov 14 | AP reports services fully restored; some users still see intermittent access issues and delayed email. A Microsoft representative explains the staged recovery: "we were forced to restore the service sequentially, checking stability at each stage … this method takes longer [but] helps ensure the system operates reliably after reopening." |
The Trigger: A Latent Fault, Exposed
Microsoft's Friday statement is the most complete public account: the outage was introduced during an earlier network capacity upgrade and surfaced after a more recent network change. The Verge's Tom Warren reported that sources link the incident to configuration changes in Azure Virtual Network, Microsoft's cloud networking service. Microsoft has not confirmed that detail. Australia's Home Affairs Minister said Microsoft told the government it was dealing with "an update that went wrong."
Note the epistemics, because they matter. During the incident, Microsoft first said it had found no recent changes correlated with the start of impact, then identified a small number of configuration updates to investigate, then reverted a network change that "correlates with the start of impact." Correlation is a lead, not a finding — rolling back anything that changed near the incident start is standard incident hygiene precisely because it is cheap and safe. The definitive account has to wait for the PIR.
What is established is the mechanism downstream of the trigger, and Microsoft documented it in its own words across the night: servers failing automated health checks, capacity concentrating, a core process becoming overwhelmed, servers unresponsive again once traffic returned. The network change is the match. The fire was the automation.
The Mechanism: A Capacity Death Spiral
Health checks are capacity guard dogs. A load balancer or orchestration layer polls each backend; a node that fails its probe is pulled from rotation before it can serve errors. The entire design rests on one assumption: the probe is a trustworthy judge. On November 12, the judge was compromised.
The loop, in Microsoft's own language: "the elevated load isn't driven by an increase in traffic, but by a number of servers being unexpectedly unavailable; straining the remaining infrastructure." Servers failed checks and were evicted. The same traffic now hit fewer servers. "A core process on the affected servers is becoming overwhelmed, which causes the servers to stop responding" — and a server that stops responding fails its next probe. Eviction begets eviction.
Run the arithmetic on a fleet cruising at 70% utilization. Lose 15% of your nodes and each survivor must run at roughly 82% to hold the same throughput. Lose 35% and each survivor needs about 108% of its capacity — which does not exist. Past 100% there is no steady state: every additional eviction pushes every remaining node further over its cliff, and the pool empties in rounds. Microsoft separately confirmed elevated processor usage on the affected fleet, which is exactly what this model predicts. Nothing about offered load changed — which is why Microsoft could say, truthfully, that this was not a traffic problem.
graph TD
T["Latent fault exposed by network change"] --> A["Nodes fail automated health checks"]
A --> B["Evicted from load-balancer rotation"]
B --> C["Same traffic, fewer servers"]
C --> D["Secure-connection process overwhelmed, CPU pins"]
D --> E["Busy survivors miss probe deadlines"]
E --> B
B --> F["Restarted node passes check, rejoins pool"]
F --> G["Retry wave forces full TLS handshakes"]
G --> D
The Bottleneck: A TLS Handshake Storm
Microsoft's most technically revealing update said that "a process that establishes secure connections was becoming overwhelmed." Combined with the elevated CPU readings, this points at the TLS handshake path — and the reason it folds into a death spiral is the asymmetry of cryptographic cost.
In steady state, traffic rides established connections using symmetric ciphers (AES-GCM, ChaCha20) that are cheap and often hardware-accelerated. A full TLS handshake is different: the key exchange (ECDHE) and signature operations are asymmetric, and cost orders of magnitude more CPU per connection than the bulk encryption that follows. Session resumption (TLS 1.3 PSK, session tickets) normally amortizes this cost away — clients that reconnect to a server they have seen recently skip most of the expensive math.
Eviction breaks that bargain. When a server leaves the pool, its clients are rehashed onto other servers. Unless session-ticket keys are shared across the fleet, every moved client performs a full handshake on a server that is already busier than usual. Now multiply: Outlook desktop and mobile clients reconnect automatically, with retries, by the millions. A reconnecting fleet is a fleet performing full asymmetric handshakes in synchronized waves — landing on the healthy pool as it shrinks. This is the classic thundering herd, except the herd is carrying a computational surcharge per request.
This is analysis, not Microsoft's published finding — but it is the only model that explains their three observed facts simultaneously: traffic did not increase, CPU was elevated, and a secure-connection process was overwhelmed. The CPU was not spent serving mail. It was spent re-establishing the privilege of serving mail.
graph LR
R["Server restarted, healthy at idle"] --> P["Passes health probe"]
P --> T["Traffic reintroduced"]
T --> W["Millions of synchronized reconnects"]
W --> X["Full TLS handshakes, CPU pinned at 100%"]
X --> F["Probe missed, evicted again"]
F --> R
Why the Fixes Kept Failing
The Register's headline — "outage stretches past 27 hours as its fixes keep failing" — is the honest summary of the mitigation log. Each failure has a mechanical explanation:
- Restarts are not free. A server restart can take up to 30 minutes depending on hardware and workload, and a post-crash disk check on large drives can extend that. An automated restart loop on a fleet of thousands is a capacity recovery measured in hours, not minutes.
- Re-entry re-runs the gauntlet. Every restarted node that rejoined the pool was fed the retry storm and had to survive the handshake wall it had just died on. Health-at-idle is not health-under-load; the fleet oscillated between those two states for most of a day.
- The fleet's own repair channel was impaired. Microsoft's 05:00 UTC update admits "machines being stuck in a bad state, preventing our changes from syncing as expected." When config distribution and change rollout are themselves unreliable, even correct fixes deliver slowly and unevenly.
- Partial recovery is not recovery. Microsoft repeatedly observed "partial recovery across regions" that was "not resulting in sustained recovery." In a feedback loop, half the capacity back still leaves every survivor over 100% — the math does not forgive half measures.
What finally worked, per the mitigation log: an upstream authentication/routing configuration component patch that stabilized the secure-connection path, the network-change rollback, an alternative health-check method, automated restart systems, and a 48-hour email retry timeout that deliberately slowed the herd by holding queued mail back from the fleet while it rebuilt. That last item deserves emphasis: Microsoft bought stability by throttling its own email ingress — an admission that the fastest way to kill the recovering fleet was to feed it its backlog at full speed.
The Precedent Nobody Should Ignore: September 30, 2026
If the November 12 pattern feels familiar, it should. Microsoft's own Post Incident Review 7Q30-010, published six weeks earlier for the September 30 Azure gateway degradation, documents the mirror image of every failure mode above:
- A paused deployment restriction expired, and "a large number of gateway updates resumed within a short period," generating a surge of restarts and configuration-retrieval requests.
- A previously deployed change had increased the processing time for those very operations in a shared regional control plane. The combination overwhelmed the control plane — and "existing request-throttling and load-protection mechanisms did not sufficiently protect critical processing under these conditions."
- Restarting ExpressRoute gateways "were marked healthy before their configuration and routing state had been fully restored" — a false-positive health check that let the updater move on and take the next redundant instance down before the first was actually ready. Connectivity loss cascaded.
- Gateways self-healed only when they gave up on the control plane and applied their locally stored last known good (LKG) configuration after about 30 minutes of failed retrievals.
Put the two incidents side by side. September 30: half-restored servers judged healthy too early — a false-positive cascade. November 12: healthy servers judged dead — a false-negative cascade. Both are failures of the same assumption, that the health-check verdict is trustworthy. Both overwhelmed the same safety net, load protection, and both were triggered by Microsoft's own change management. Two sides of the same coin, six weeks apart, in the same company. Health checks are the most safety-critical and least-audited component in most fleets.
The Pattern: This Is What Microsoft Outages Are
An analysis of 79 Azure Post Incident Reviews published between November 2021 and September 2026 finds that 44 of 79 (55.7%) began with a change Microsoft made itself — a configuration change, a code deployment, an automated key or certificate rotation, or a maintenance and decommissioning tool. Those self-inflicted incidents account for 51.3% of all customer impact hours. The median Azure incident ran 8.6 hours.
The November 12 outage outran 54 of those 79 incidents by Friday morning, and kept going. Recent PIRs in the set read like a genre: a December 2025 automated key rotation broke Azure Resource Manager in China and later Government; a November 2024 decommissioning process "inadvertently deleted" Traffic Manager profiles; a July 2024 Azure OpenAI change "took effect soon after it was committed" and deleted resources for 23.6 hours of impact. An expert quoted by Ars Technica suggested this outage may end up rated worse than Amazon's or CrowdStrike's 2024 events. When your failure signature is this consistent, the problem is not bad luck — it is the architecture of change.
The Documented Blast Radius
This was not "email being slow." Exchange Online is the control plane of a remarkable amount of Western civilization's paperwork, and the incident log reads like a monoculture stress test:
- Courts: A criminal trial in Manchester was suspended with the jury sent home; staff reportedly hunted for a fax machine. The judge remarked that the episode illustrated "the danger of the court's reliance on technology." Maine's judicial branch warned that emailed requests for protection orders filed after 12:30 p.m. Thursday might never have been received. New York City suspended evictions scheduled for Friday. The UK Supreme Court and the Supreme Court of the Philippines both reported losing email. In Colorado, the Arapahoe County sheriff said the jail could not process online bond payments — immediate releases required in-person payment.
- Healthcare: Unity Health Toronto declared a "Code Grey" (infrastructure failure). SickKids, Canada's largest children's hospital, told families email and phone contact with staff might fail and directed emergencies to the nearest ED. NHS trusts across England reported staff email outages; the National Pharmacy Association warned of prescription delays because doctors could not email prescriptions to pharmacies.
- Police and government: UK police forces stopped receiving online crime reports and urged the public to phone instead. Municipal email went down in Austin, Jacksonville, Baltimore and elsewhere. Japan, Singapore and Australia reported government operations affected. DOJ attorneys filed for deadline extensions because the outage prevented "completing the necessary coordination"; a US congressional staffer quipped that the Legislative Branch was "currently paralyzed by the Outlook outage."
- Markets and business: Euronext, the European stock market operator, said it was unable to receive emails. Banks in Hong Kong saw disruptions; a Hong Kong aluminum exporter lost contact with US customers; UK law firms warned that Friday — the standard house-completion day — put property transactions at risk.
- Second-order effects: Fortune's analysis is the one to internalize: email is also the recovery channel for everything else. Password resets and verification codes stopped arriving, locking people out of unrelated services — banking, payments, document signing. The University of Sussex warned students of phishing attempts posing as Microsoft engineers offering to "fix" email. An outage of your identity-adjacent systems multiplies into an outage of everything that depends on them to bootstrap.
The societal dependency is the real finding. One vendor's configuration change, applied to one service tier, suspended trials, evictions, bond postings, prescriptions and a stock exchange's inbox within a single working day. That is not an availability problem. That is a monoculture problem.
The Prescription: Nine Controls That Would Have Contained This
Every item below maps to a specific, documented moment in the incident. These are not aspirational — they are standard, boring controls that this fleet demonstrably lacked:
- Panic / fail-open thresholds. Envoy's healthy panic threshold (default 50%) exists for exactly this scenario: if fewer than half the pool is healthy, ignore health status and spread load across everyone anyway. A slow answer beats no answer. This single control converts a total outage into a brownout.
- Split liveness from readiness. Kubernetes keeps these probes separate for a reason: "process is alive" and "process wants traffic right now" are different questions. Evicting on readiness flapping alone is what turns transient latency into permanent capacity loss.
- Protected probe paths. A probe that shares a thread pool, CPU budget, or TLS stack with user traffic fails precisely when the server is busy — measuring load, not health. Probes must be cheap, prioritized, and independent of the path they judge.
- Cap the eviction rate. Automation must never be able to remove more than a bounded percentage of a pool per interval, no matter what the probes say. On November 12 this is the difference between "degraded capacity in some regions" and "72 countries on Downdetector." There is no evidence any such cap existed.
- Make session resumption survive churn. Share TLS session-ticket keys across the fleet, or terminate TLS at a tier that does not churn with the backend pool. Otherwise every eviction converts every displaced client from a cheap resumed session into a full asymmetric handshake.
- Jittered exponential backoff in every client. Synchronized retry waves are a self-inflicted DDoS. Any client fleet — and Microsoft has hundreds of millions — must de-synchronize on retry.
- Cell-based architecture. Independent capacity cells (shards, regions, stamps) with independent failure domains confine a feedback loop to the cell where it started. A failure in one cell stays in that cell; the November loop went continental within minutes.
- Static stability. Nodes must keep operating on last-known-good configuration when the control plane lies, dies, or stops syncing. Microsoft's own September gateways did exactly this (LKG after 30 minutes) — apply that pattern to the fleet, not just the gateways.
- Treat health checks as safety-critical code. Version them, canary them, audit their eviction decisions, and load-test them at several times the normal eviction rate. The component that can delete your entire fleet in minutes deserves at least the review rigor of the code that serves one request.
Who Should Skip This Stack
The answer is not "leave Microsoft." The answer is: stop depending on any single vendor's availability for processes you cannot run degraded. Concretely:
- Skip single-vendor email if email is your recovery channel. If password resets, court filings, prescriptions, or bond postings arrive exclusively through one vendor's Exchange Online, your incident recovery plan has a single point of failure with a 27-hour median repair time and a 55.7% chance of being self-inflicted.
- Skip cloud-native identity monoculture without a break-glass path. When the IdP-adjacent tier fails, every dependent service fails, however healthy it is. Courts and hospitals learned this on November 13: the workaround was fax machines, phone lines and paper — whatever your organization's equivalent is, it needs to exist before the outage.
- Skip "we'll handle it during the incident" capacity planning. Microsoft's own throttling did not protect the control plane on September 30, and its health checks did not protect capacity on November 12. If the vendor's automation cannot reliably protect the vendor, your automation must protect you.
When the PIR lands, we will check it against this reconstruction — in particular whether the Azure Virtual Network report is confirmed, whether an eviction-rate cap is among the promised defense-in-depth actions, and whether the health-check component that "processes server health" is being redesigned or merely patched. Until then, treat the nine controls above as the checklist this incident wrote for everyone else's platform.