Rehearse the Disaster Before It Happens

The Decision in Front of You

A disaster recovery plan states how long each critical service may be down and how much data the business can afford to lose. Those targets go into contracts, board papers, and audit responses. Until someone has restored the service within the target, the numbers are estimates.

The decision for operations and technology leaders is whether to spend a few days of staff time each year proving the plan works, or to let the first real outage do it. Exercises cost money and attention, and the cases below show what the alternative cost two large organizations.

NIST's guidance on testing IT plans separates three activities ¹². Training tells people their roles. Exercises simulate an emergency to check whether the plan holds up. Tests use measurable results to confirm that a system or component works as the plan says. A recovery program needs all three.

What's Happening

Two recent cases where recovery set the cost

Delta Air Lines, July 2024. On July 19, 2024, a faulty security software update crashed Windows systems at airlines, banks, and hospitals worldwide. Delta cancelled about 7,000 flights over five days, affecting more than a million customers, while United cancelled about 1,500 ⁷. Delta's chief executive put the cost at about $500 million ⁶. The crew-tracking system that matches pilots and flight attendants to flights was crippled, and the airline had to reset 40,000 servers by hand ⁶⁷. Delta's pilot and dispatcher unions said the crew scheduling system dates from the 1980s ⁷. The U.S. Department of Transportation opened an investigation because Delta's recovery lagged its peers, and coverage in June 2026 reported the inquiry closed without major fines ⁸.

Delta's competitors were hit by the same update on the same morning ⁷. What separated them was how fast each could rebuild its operation. For Delta, the bottleneck was a downstream business system that had to be repopulated and resynchronized after the computers came back. Public reporting does not say whether Delta had exercised a scenario like this one.

South Korea's government data center, September 2025. A battery fire on September 26, 2025, at the National Information Resources Service in Daejeon destroyed 96 systems and disrupted 647 ¹¹. Data submitted by the office of a National Assembly member showed that only 28 of the 647 systems at the site, 4.3%, had server-level disaster recovery that could switch to a standby server. Only 19 had replicated storage. None of the 96 destroyed systems had server-level recovery ¹⁰. A shared document store used by government staff had no external backup at all, and its contents were reported as permanently lost ⁹. In early October, 115 of the 647 systems were back, about 18%, and officials expected full recovery to take a month ¹¹.

Almost all of the destroyed systems had backups ⁹¹¹. What most of them lacked was a fast path back. A failover drill for each critical system would have produced a written list of the ones that could not fail over, while there was still time to fund the fixes.

Exercises expose what documents hide

Change Healthcare's 2024 ransomware attack is a third example. Its chief executive later told Congress that the primary and backup environments were not isolated from each other, so both were locked at once, and that cloud-hosted systems came back far faster than older data center systems ¹². A drill that tried to recover the backup environment without touching production would have shown the shared exposure.

Each case has the same shape. The plan assumed something would be available on the day: a scheduling system, a standby server, a separate backup environment. Nobody had proven it under realistic conditions.

Standards and regulators expect proof

NIST SP 800-84, the federal guide to test, training, and exercise programs for IT plans, calls for documenting each event in an after-action report and repeating exercises after organizational changes and plan updates ¹². Under NIST SP 800-53, federal agencies must exercise or test contingency and incident response plans at least once a year ². The CISA #StopRansomware Guide asks organizations to exercise their incident response plan regularly, keep hard copy and offline versions, and have the chief executive approve the plan in writing ⁵.

ISO 22301:2019, the international standard for business continuity management, requires exercises at planned intervals, based on realistic scenarios, that confirm the continuity arrangements work, produce formal post-exercise reports, and lead to improvement actions ¹³. The FFIEC Business Continuity Management booklet, which replaced the earlier planning booklet in November 2019 and guides US bank examiners, treats testing as one of the core areas of an institution's continuity program ¹⁴. In the European Union, the Digital Operational Resilience Act has applied to financial entities since January 17, 2025. It requires a resilience testing programme and testing at least once a year of the ICT systems that support critical or important functions ¹⁵. Larger firms named by their supervisors must also run threat-led penetration testing at least every three years ¹⁶.

Why the Usual Approach Fails

Plans are written for an audience of auditors. The document describes the target state of recovery in careful language, gets signed, and goes into a repository. Nobody reads it until the event, and by then some of the names in the contact list belong to people who have changed jobs.

Tests are scoped to pass. Restoring one virtual machine from last night's backup proves the backup software works. It says nothing about restoring 200 servers in the right order while the identity system is down and the network team is on a bridge call with a vendor. Teams pick the easy test because a failed drill feels like a failing grade.

Tabletops stay in the conference room. A discussion exercise is valuable for decision-makers, and NIST is explicit that a tabletop deploys no equipment or resources ². It cannot reveal that a restore takes 14 hours instead of the four the plan promises. Only a functional drill can measure that.

The business process is left out. In the Delta case, the computers came back before the operation did ⁶⁷. Recovery plans written by IT usually end when the servers are up. Customers experience the end of the outage when orders ship, crews fly, or claims pay.

The plan lives inside the system that failed. If the runbook, the contact list, and the passwords sit on a file share that is encrypted or offline, the team starts the recovery from memory. CISA recommends keeping printed and offline copies for this reason ⁵.

Findings do not get fixed. An exercise that produces a list of gaps with no owners and no dates leaves those gaps in place for the next real event.

The Framework: The Exercise Ladder

Each type of exercise proves something the one below it cannot. A mature program climbs all the rungs every year and moves higher only after the lower rung runs cleanly.

The cycle around each exercise matters as much as the exercise.

flowchart LR
    A[Choose scenario<br/>and success<br/>measures] --> B[Run exercise<br/>record times<br/>and decisions]
    B --> C[After-action<br/>report within<br/>two weeks]
    C --> D[Assign each<br/>finding an owner<br/>and a date]
    D --> E[Re-test the<br/>fixed items]
    E --> A

Choose the scenario. Pick events the business has reason to fear: loss of a data center, ransomware in the directory service, a failed vendor update, a key supplier going dark. CISA publishes more than 100 tabletop exercise packages with ready-made scenarios, facilitator guides, discussion questions, and after-action report templates ³⁴. Set success measures before the exercise starts, using the recovery targets in the plan.

Run it with the business present. Invite the people who run the process that depends on the system: dispatch, customer service, finance, crew or shift scheduling. The exercise ends when the business process is running again, which can be well after the servers are.

Write it down. A short after-action report records what happened, how long each step took against the target, and what went wrong ¹².

Fix and re-test. Every finding gets an owner, a due date, and a re-test. The report to leadership shows open findings by age, the same way audit findings are tracked.

Costs, Risks, and Tradeoffs

How to Act

Questions to Ask Your Team

1. For each critical service, when did we last restore it in a drill, and how long did it take against the target?
2. Which services have no tested recovery path at all?
3. If the file share and email were down, where would the team find the plan and the contact list?
4. Which business systems, like scheduling or order management, have to be rebuilt after the servers come back, and is that time in our target?
5. What did our last exercise find, and how many of those findings are still open?
6. Who outside the recovery team watched the last drill?
7. Which outside vendors would we need on the phone in the first hour, and have they ever joined an exercise?

Closing

A recovery plan describes a capability the company hopes to have, and exercises measure how much of it exists. Climb the ladder each year, put the business process owners in the room, write down what broke, and fix it before the next drill. Report measured recovery times beside the targets, so leadership can see the distance between the plan and what the team can deliver on its worst day.