Build High Availability Before Disaster Recovery

The Decision in Front of You

Every resilience budget forces a choice about what to protect against first: losing the whole primary site, or losing one piece of it.

Protection against each pays back on a different schedule. High availability, meaning a system built with enough redundancy to keep serving through a component or zone failure without human intervention, earns its keep whenever a disk, host, switch or zone fails. Disaster recovery, meaning the ability to restore service in a different location after the primary one is lost, earns its keep in the rare event that the whole site or region goes.

The money at stake is material. In Uptime Institute's 2026 outage analysis, 57% of operators said their most recent major outage cost more than $100,000, and one in five put it above $1 million for the second year running ¹.

What's Happening

Most outages are local

Uptime's 2026 report found power caused 45% of the most recent impactful outages in 2025, led by failures of uninterruptible power supplies, transfer switches and generators ². Network problems are the most frequent cause of IT service outages, and configuration and change management remain the main source of those ². Within IT systems, firmware and software faults now account for the largest share ². A year earlier, IT and networking issues had risen to 23% of impactful outages, with change management and misconfiguration the likely cause ¹⁰.

These are local failures: a UPS in one room, a misconfigured switch, a database host that loses a disk. Redundancy and automatic failover are designed to absorb exactly these.

Uptime also found human error contributed to nine of ten outages, and 87% of operators with an impactful outage said better management, processes or configuration could have prevented it ². In the 2025 analysis, 85% of human-error outages traced to staff not following procedures or to flawed procedures ¹⁰.

The standards already say so

NIST SP 800-34 Rev. 1, the federal contingency planning guide, scopes a disaster recovery plan narrowly: it covers information system disruptions that require relocation to an alternate site ³. The same guide says strategies for high-impact systems should consider high availability and redundancy in their design, and that preventive measures are preferable to recovery actions where they are feasible and cost-effective ³.

ISO 22301:2019, the international standard for business continuity management systems, frames continuity as reducing the likelihood of disruption as well as recovering from it ⁴. Amazon's Well-Architected guidance tells customers to run every production workload in at least two availability zones, separate data centers with independent power, cooling and networking within one region. It rates single-zone production as high risk and reserves multi-region designs for workloads that need extreme resilience ⁵.

Two 2026 cases show the difference

A single zone overheats, May 2026. On May 7, 2026, multiple cooling units failed in one AWS data hall in Northern Virginia, and racks shut down to protect themselves ⁶. Coinbase lost trading for several hours, and full recovery took much of the following day ⁶. Its postmortem found the matching engine ran as a five-node cluster packed into one placement group to cut latency. The thermal event took three of the five nodes, the cluster lost quorum, and the design had no automated failover to another zone ⁶. Coverage of the postmortem gives no cost figure.

The lesson is about high availability. The failure was confined to one zone, and the customer's architecture had concentrated a critical system inside it. Coinbase's remediation list starts with automated cross-zone recovery and adds expanded disaster recovery testing ⁶.

A region is struck, March 2026. On March 1, 2026, objects struck an AWS facility in the United Arab Emirates during the conflict between the United States and Iran, causing a fire. Firefighters cut power to the building and its generators, and a second zone in the region also lost power ⁷⁸. Months later, AWS said it still could not restore resources and data held only in the struck zone, and it advised customers to move to other regions; most did so by restoring backups or copying data that remained reachable ⁸.

This is the event disaster recovery exists for. Multi-zone design could not save a workload when two zones in one region failed together, and data kept only in the struck zone was still out of reach months later. Copies in another region let customers resume.

Software can take a region down too: the AWS DynamoDB DNS failure of October 2025 hit most services in Northern Virginia for up to 15 hours ⁹.

Why the Usual Approach Fails

Disaster recovery is easy to buy and easy to audit. A secondary site contract and a binder of runbooks are evidence that a plan exists, and auditors and insurers ask for that evidence, so it gets funded.

High availability gets little scrutiny. Few audits check whether the order database has a replica in a second zone. Engineers make those choices under delivery pressure, and the cheap option is a single instance in a single zone.

The company ends up owning a secondary site it has never failed over to under real conditions, while single points of failure in the primary environment cause most of its outages. When a local failure hits, the team will not declare a disaster for a problem that might be fixed in an hour, so the DR investment sits unused while the outage runs for three.

Failover that runs only during an annual test also decays between tests. Automatic failover within a region runs whenever something breaks, so the team learns quickly whether it works.

The Framework: The Blast Radius Ladder

Rank each failure by how often it happens and how far it spreads. Fund protection rung by rung, and stop climbing where the next rung costs more than the risk it removes.

flowchart LR
    A[Rung 1: Component<br/>disk, host, process<br/>Weekly to monthly] --> B[Rung 2: Zone or data hall<br/>power, cooling, network<br/>Yearly]
    B --> C[Rung 3: Bad change<br/>config or deployment<br/>Monthly]
    C --> D[Rung 4: Region or site<br/>fire, flood, conflict, regional software fault<br/>Once in many years]
    D --> E[Rung 5: Data loss<br/>corruption, deletion, ransomware<br/>Any time]

Rung 1, components. Run at least two of everything that serves customers, behind a load balancer or a clustered database with automatic failover.

Rung 2, zones. Spread each critical service across at least two zones or two independent rooms, with data replicated synchronously ⁵ and failover tested monthly. Coinbase shows the failure mode: redundancy that lives inside one zone counts as rung 1 protection ⁶.

Rung 3, changes. Roll changes out one zone at a time with an automatic halt and rollback. Uptime's data puts configuration and change at the center of network outages ².

Rung 4, regions and sites. Here disaster recovery begins. Choose the cheapest pattern that meets the recovery time objective, or RTO, the maximum tolerable time to restore service, which NIST ties to the business's maximum tolerable downtime ³. Many systems need only a pilot light, a minimal copy of the environment that scales up when needed. A few need a warm standby, a smaller running copy ready to take traffic ⁵.

Rung 5, data. Immutable backups in a separate location apply to every critical system regardless of the other rungs. After the UAE strike, restoring backups is how most affected customers moved to other regions ⁸.

What the ladder is worth

The following is an illustrative model for a hypothetical distributor whose order platform brings in $120,000 per hour. Substitute your own figures.

High availability returns its cost several times over on events that happen every year. Disaster recovery value swings with the probability of a regional event and the pattern chosen, so it has to be sized deliberately. Permanent data loss cannot be priced per hour, which is why rung 5 applies to every critical system.

Costs, Risks, and Tradeoffs

How to Act

Questions to Ask Your Team

1. Of our last ten outages, how many would a second zone or a redundant component have prevented, and how many needed a second region?
2. Which critical systems still run as a single instance or in a single zone?
3. When did we last shut down a zone on purpose, and what happened?
4. Do our deployments reach every zone at once, or one at a time?
5. What does an hour of downtime on each critical system cost, and who agreed to that number?
6. How many times have we used our DR capacity for a real event?
7. Could an attacker with production credentials delete our backups?

Closing

Disaster recovery covers the day the building or region is gone. High availability covers the far more common day when a server, a zone or a change fails. Spend first on the architecture that absorbs frequent local failures, because it pays back every year. Then buy disaster recovery sized to the regional risk that remains, and keep immutable backups under every critical system regardless.