Every Change Should Be Reversible

The Decision in Front of You

Engineers ship code, administrators edit settings, and vendors push updates to your laptops and servers every day. The decision is whether each of those changes can reach every customer at once, or whether it must earn its way out in stages with a working undo button.

The stakes sit in a few bad days. Uptime Institute's 2026 outage report found that 57% of operators said their most recent major outage cost more than $100,000, and one in five put the cost above $1 million ¹. In Uptime's 2025 analysis, IT and networking problems accounted for 23% of impactful outages in 2024, a rise Uptime tied to growing complexity and the change-management mistakes and misconfigurations that come with it ².

Speed is also climbing. Google's 2025 DORA survey of nearly 5,000 technology professionals found that AI adoption now raises software delivery throughput and still has a negative relationship with delivery stability ³. The report's authors warn that more change volume produces instability unless teams have strong automated testing and fast feedback loops ³.

What's Happening

The pattern behind the largest recent outages

Each of the following failures began with a deliberate change that spread everywhere before anyone could judge whether it was safe.

CrowdStrike, July 2024. On July 19, 2024, CrowdStrike released a content configuration update for its Falcon sensor on Windows ⁴. A new template expected 21 input fields, the sensor code supplied 20, and the mismatch caused an out-of-bounds memory read that crashed the machine ⁴⁵. A logic error in the content validator let the file through, and this category of content had not faced the same tests and reviews as sensor software ⁵. Microsoft estimated about 8.5 million Windows devices were affected, under 1% of all Windows machines ⁶.

The cost landed on CrowdStrike's customers. The insurer Parametrix estimated $5.4 billion in direct losses for the US Fortune 500 excluding Microsoft, with 125 companies affected, including all six airlines on the list and roughly three quarters of the healthcare and banking firms ⁷. Parametrix put insured losses at only $0.54 billion to $1.08 billion ⁷. Delta Air Lines told the SEC it expected about $550 million in losses, after canceling roughly 7,000 flights over five days ⁸.

CrowdStrike's fixes include deployment rings, canary testing before wider release, rollback when problems appear, and customer control over when content updates land ⁴⁵.

Google Cloud, June 2025. In May 2025, Google rolled out new quota-checking code in Service Control, which authorizes API requests across Google Cloud ⁹. The rollout went region by region, but the new code path ran only when a certain kind of policy changed, so the staged release never exercised it. The code lacked error handling and had no feature flag, a switch that turns new behavior on for a few customers at a time ⁹.

On June 12, 2025, a policy update with blank fields landed in a database that replicates globally within seconds, and Service Control crashed in a loop in every region ⁹. Engineers found the cause in about 10 minutes. Most regions still took around two hours to recover, and one large US region took close to three as restarting services overloaded the database behind them ⁹. Google's report says a feature flag would have caught the problem in staging, and its fixes require flags, off by default, on critical binaries ⁹.

Microsoft Azure Front Door, October 2025. On October 29, 2025, a tenant configuration change in Azure Front Door, Microsoft's global edge routing service, left many edge nodes unable to load ¹⁰. Safeguards meant to block faulty deployments were bypassed because of a software defect, and at least 20 Azure services were affected, including the Azure Portal ¹⁰. Microsoft's post-incident summary, as relayed by affected customers, puts the impact from 15:41 UTC on October 29 to 00:05 UTC on October 30, about 8 hours 24 minutes ¹¹. Recovery meant freezing configuration changes and pushing the last known good configuration worldwide ¹⁰.

Cloudflare, December 2025. On December 5, 2025, a configuration change made to shield customers from a new software vulnerability caused 28% of applications on Cloudflare's network to fail for about 25 minutes ¹². Its configuration system reached 90% of servers within seconds, while its software releases already moved through staged gates with automatic rollback ¹². In 2026 it finished applying the same staged, health-checked rollout and automated rollback to configuration changes ¹³.

What the measurements say

DORA's 2024 research grouped teams into performance clusters. The top group saw about 5% of changes fail and recovered from a failed deployment in under an hour. The bottom group saw about 40% fail and took between one week and one month to recover ¹⁴. A team at the bottom fails eight times as often and takes far longer to undo each failure.

In Uptime's 2025 report, nearly 40% of organizations had suffered a major outage caused by human error in the prior three years, and 85% of those incidents came from staff not following procedures or from flawed procedures ². The share tied to skipped procedures rose ten points in 2025 ². A staged rollout with automatic rollback is a procedure the system enforces even when a tired person skips a step.

Regulators are already writing this down. The EU's Digital Operational Resilience Act technical standard on ICT risk, Commission Delegated Regulation (EU) 2024/1774, requires financial entities to plan and test changes in a controlled way, keep approval independent from the people requesting and implementing a change, and define fall-back procedures and responsibilities, including for aborting a change ¹⁵. Emergency changes must be documented and re-assessed after the fact ¹⁵.

Why the Usual Approach Fails

Most companies already have change control, usually a ticket, a weekly approval meeting, and a maintenance window. That process was built for large, infrequent software releases, and recent outages started elsewhere.

Configuration and data move faster than code. In three of the four cases above, the harmful change was a setting, a policy record, or a content file. Each traveled a fast path built to save minutes on good changes, with no stage where a bad one could be caught small ⁵⁹¹².

Approval is a weak test of safety. A review meeting confirms that a change looks sensible on paper, and the defects that matter tend to appear only under live traffic. Google's own staged rollout passed the faulty code region by region because nothing triggered it ⁹. Exposure to a small amount of real traffic, watched closely, is what catches them.

Rollback is assumed and seldom practiced. Google's site reliability engineers have advised since at least 2017 that rollback should be the first response to a suspected bad release, with diagnosis afterward, and that teams should roll back periodically just to prove they still can ¹⁶. Untested rollback makes engineers hesitate, and the outage grows while they try to fix forward.

Vendor changes sit outside the process entirely. CrowdStrike's customers had no say over when the faulty content arrived on their machines ⁴. Many companies still accept vendor auto-updates with no ring of their own and no contractual right to delay.

The Framework: The Reversibility Gate

A safe change reaches a small, representative slice of production first, automated health checks compare that slice against the rest, and the change reverses itself if the checks degrade, without waiting for a human decision. Google's production guidance describes the same structure: nonemergency rollouts proceed in stages, each stage is supervised by monitoring, and unexpected behavior triggers rollback first and diagnosis second ¹⁷.

flowchart LR
    A[Change<br/>code, config,<br/>data, or vendor] --> B[Validate<br/>schema and<br/>sanity checks]
    B --> C[Canary<br/>one percent<br/>or one site]
    C --> D[Health<br/>signals<br/>steady]
    D -->|Yes| E[Widen<br/>in rings<br/>with pauses]
    D -->|No| F[Automatic<br/>rollback to<br/>last known good]
    E --> D
    F --> G[Review<br/>before<br/>retrying]

Validate. Reject malformed input before it ships. A file with an unexpected field count or blank values should fail in seconds. When a system receives implausible configuration, it should keep running on its previous state and raise an alert ¹⁷. Cloudflare now calls this "fail stale" ¹³.

Canary. Send the change to a small slice first: one percent of traffic, one site, one internal team. Cloudflare starts with employee traffic, then free customers, before paying ones ¹².

Watch. Compare the canary's error rates, latency, and crash counts against the baseline automatically, using signals defined before the change ships.

Widen or reverse. If signals hold, widen in rings with pauses between them. If they degrade, roll back automatically to the last known good version. A person reviews the failure before anyone retries.

The arithmetic of blast radius is simple. For a hypothetical distributor with 40 sites, a bad change pushed everywhere and fixed in four hours costs 160 site-hours of disruption. Sent to one site and reversed by health checks in 15 minutes, it costs a quarter of a site-hour, 640 times less. This is the author's illustrative estimate; it assumes the canary site is representative and the health checks detect the fault.

Costs, Risks, and Tradeoffs

Reversibility costs engineering time and slows the fastest changes. Some changes cannot be reversed at all.

How to Act

Questions to Ask Your Team

1. Which of our production systems can change everywhere at once, and who can make that change?
2. Do configuration changes and database changes go through the same staged pipeline as code?
3. What share of our changes failed last quarter, and how long did each take to undo?
4. When did we last roll back a production change on purpose, and how long did it take?
5. Which vendors can push software or content to our servers and laptops without our approval?
6. For our five most important systems, which health signals would trigger an automatic rollback?
7. Which changes cannot be reversed, and what extra checks do they get?

Closing

The outages that cost the most in recent years started with routine changes released to everyone at once. A change that reaches one percent of customers and reverses itself costs minutes. The same change at full scale cost Fortune 500 companies billions in a single day in 2024 ⁷. Require every change, including configuration and vendor updates, to prove itself on a small slice and to undo itself when it fails, and ask to see the change failure rate alongside revenue.