Recurring Incidents Are Unpaid Debt

The Decision in Front of You

Engineering closes incidents every week. In many companies a closed ticket means service came back through a restart, a rollback, or a manual workaround. The defect or unclear procedure that caused the failure is still there, unbooked and absent from any forecast.

The decision for a CFO and CTO is whether to fund root-cause work as a standing obligation with a budget, an owner, and a monthly balance, or to leave it to whatever time the feature roadmap allows. In Uptime Institute's 2026 survey, 57% of respondents said their most recent major outage cost more than $100,000, and one in five put it above $1 million ¹. A failure class that repeats at those figures is a material expense no budget line names.

What's Happening

Most repeat failures have known, fixable causes

Uptime's 2025 analysis found nearly 40% of organizations had a major outage caused by human error in the previous three years. Of those, 85% traced to staff not following procedures or to flaws in the procedures themselves ². The share caused by not following procedures rose ten percentage points from the prior year ², and in 2026 it remained the leading driver of human-error outages ¹. Coverage of the 2025 report noted that about 80% of operators believed better management and processes might have prevented their most recent major outage ³.

The engineering literature already treats recurrence as the core risk

Google's SRE book lists events that should trigger a written postmortem, such as user-visible downtime beyond a threshold or any data loss. It warns that without a formal learning process, incidents may recur indefinitely, and it says an unreviewed postmortem might as well never have existed ⁴.

Google's example error budget policy turns this into deadlines. If a single incident consumes more than 20% of a service's four-week error budget, the postmortem must include at least one top-priority action on the root cause. If a class of outage consumes more than 20% of the budget over a quarter, a top-priority item goes into the next quarter's plan ⁵.

The SRE book defines toil as manual, repetitive operational work that grows with the service and leaves no lasting improvement, and sets a goal of keeping it below 50% of each engineer's time ⁶. Every repeat incident handled by hand is toil.

NIST SP 800-61 Rev. 3, issued in April 2025, places incident response inside overall cybersecurity risk management under CSF 2.0, with the aim of reducing both the number and the impact of incidents ⁷.

Delivery research measures the drag

Google's DORA program added rework rate in 2024: the share of deployments that are unplanned and made to fix production problems ¹⁶. It shows how much engineering capacity goes to paying interest. In DORA's 2024 performance clusters, the lowest group had a change failure rate of 40% and took between a week and a month to recover from a failed deployment, while the top group failed 5% of the time and recovered in under an hour ¹⁷. DORA's 2025 report found AI adoption still associated with lower delivery stability, especially where testing and feedback loops are weak ¹⁵.

Recent Cases: The Interest Comes Due

North Hyde substation and Heathrow, 2025

On March 20, 2025, a fire at the North Hyde electrical substation in west London cut power to 71,655 customers ¹⁴. Heathrow Airport closed for about 18 hours and more than 1,000 flights were cancelled ¹³.

The GB National Energy System Operator's final report found the probable root cause was moisture entering a high-voltage bushing on a supergrid transformer ¹². Oil samples in July 2018 had shown an elevated moisture reading in that bushing. The operator's own guidance said such a reading indicated an imminent fault and that the bushing should be replaced. The reading was logged, the matching mitigation was never carried out, and controls did not catch that nothing had followed. In 2022, basic maintenance on the transformer was deferred ¹². NESO's first recommendation addressed asset management controls to categorize and follow up identified issues ¹⁴.

The warning sat in a system of record for almost seven years with a recorded severity, and no control forced the action that severity called for. Finance would call that an unrecorded liability.

Optus triple zero outages, 2023 and 2025

On November 8, 2023, a nationwide Optus outage lasted about 12 hours and affected more than 10 million services ⁹. The Australian Communications and Media Authority found 2,145 people could not reach emergency services and 369 welfare checks were not carried out, and Optus subsidiaries paid more than A$12 million in penalties ⁸. A government review made 18 recommendations, all accepted ⁹.

On September 18, 2025, a scheduled firewall upgrade in South Australia blocked emergency calls across several states for about 14 hours ¹⁰. The independent review found the change instructions given to the contractor were incorrect, despite 15 similar upgrades in previous months. Operations alerts got only cursory attention, and the call center did not escalate customer reports. It took about 13 hours to understand the problem ¹⁰. Hundreds of emergency calls failed, and the outage was linked to two deaths ¹⁰¹¹. The review found "gaps in process, accountability, and escalation" and made 21 recommendations, which the board accepted ¹⁰. In July 2026 the ACMA began Federal Court proceedings alleging 1,005 contraventions, each carrying a maximum penalty of A$250,000 ¹⁸.

Both outages left people unable to reach emergency services, and the second review still found gaps in process, accountability, and escalation. Penalty exposure moved from about A$12 million paid to a theoretical maximum above A$251 million, alongside the human cost.

Why the Usual Approach Fails

Restoring service is measured and praised. Removing a cause shows up only as incidents that never happen, which no dashboard counts.

Postmortem action items land in a tracker next to feature requests, with no priority, no due date, and nobody checking whether they closed. At North Hyde, the reading was in the right system and the follow-up control was missing ¹².

Two incidents eight months apart, handled by different on-call engineers, rarely get linked. Without grouping by failure class, one unpaid cause looks like isolated bad luck.

A review that ends with "the engineer made a mistake" stops short of the procedure that allowed it, which is where Uptime's data places most human-error failures ².

The Framework: The Incident Debt Ledger

Book each incident without a verified fix as a liability. The principal is the cost to remove the cause. The interest is the expected annual cost of recurrence: events per year times cost per event. Repayment is a funded action with an owner and due date. Write-off requires signed risk acceptance by a named executive.

flowchart LR
    A[Incident<br/>service restored] --> B[Blameless<br/>postmortem<br/>cause and class]
    B --> C[Book liability<br/>principal = fix cost<br/>interest = recurrence cost]
    C --> D[Repay<br/>funded action<br/>owner and due date]
    C --> E[Write off<br/>signed risk<br/>acceptance]
    D --> F[Verify<br/>no recurrence<br/>for 90 days]
    F -->|recurs| C

Book it. Every incident that meets a postmortem trigger ⁴ gets a ledger entry with a failure class, so repeats link to the same line.

Price it. Estimate cost per event from revenue at risk, response labor, credits, and penalties, and frequency from history. Revise both after each recurrence.

Fund it. Reserve a fixed share of each team's capacity for repayment before feature planning. Google's 20% thresholds are a model for when a failure class displaces planned work ⁵.

Verify it. An entry closes only after the failure class has not recurred for an agreed period. A recurrence reopens it at higher priority.

What one unpaid incident class costs

This illustrative model covers a hypothetical distributor whose order integration queue stalls several times a year.

Most budgets miss the year-two increase, as an unfixed cause grows costlier with the business.

Costs, Risks, and Tradeoffs

How to Act

Questions to Ask Your Team

1. Which incidents in the last year were repeats of an earlier failure, and what did the repeats cost?
2. How many postmortem action items are open, and how many are past due?
3. Which known warnings, test findings, or audit items have been logged with no completed action?
4. What share of last quarter's deployments were unplanned fixes for production problems?
5. Who can accept the risk of leaving a known cause in place, and where is that acceptance recorded?

Closing

An incident closed without its cause removed is a liability with interest, and the interest compounds as the business grows around the failing system. Book each one, price it, fund its repayment from reserved capacity, and review the balance monthly with finance. Close an entry only when the cause is verified gone or an executive has signed for the risk.