Automate the Process You Actually Have

The Decision in Front of You

Every operations leader now has a queue of automation proposals: bots for accounts payable, AI agents for customer service, rules engines for eligibility and claims. Each comes with a process map and a savings figure. The decision is whether to fund against that map or to first require evidence of how the work is performed today.

Automation that matches the real process removes cost and delay. Automation built on an idealized version handles the easy cases, pushes the hard ones to people who no longer have the time or context, and at worst makes the wrong decision thousands of times before anyone notices. The cost of finding out is a few weeks of discovery. The cost of not finding out has, in public cases, run to hundreds of millions of dollars and a government system taken offline for repair.

The rule this paper argues for: no automation or AI funding for a workflow without a measured baseline of the work as done, which later becomes the yardstick for whether the project worked.

What's Happening

When the rule changes and the code does not

Australia's Targeted Compliance Framework, 2022 to 2025. Australia's employment department runs an automated system that enforces the obligations job seekers must meet to receive income support. In 2022 the law was amended so that a human decision-maker had to consider an individual's circumstances before cancelling a payment ¹². That discretionary step was never built into the system. When a failure was recorded, the payment was cancelled automatically ².

The Commonwealth Ombudsman reported in August 2025 that these cancellations were unlawful, in a report titled "Automation in the Targeted Compliance Framework: When the Law Is Changed but the System Isn't" ³. The error ran from April 2022 until September 2023, when outside lawyers advising on an unrelated matter noticed it ². The department took almost ten more months to pause cancellations, which the Ombudsman called unacceptable ¹. The report found the people who drafted the law change assumed it needed no rework of systems or processes, and that the drafters were wrongly assumed to understand how the automated system worked ². A separate review found the system's IT had become unstable and identified five errors dating back to 2018 ². In mid-2026 the department told a Senate committee the system would stay offline for several months for changes to IT systems, procedures and training, with 964 job seekers affected ⁴.

Even the review that confirmed the problem needed correcting: Deloitte agreed in October 2025 to partially refund its roughly A$440,000 fee after its report on the framework was found to contain fabricated references, produced partly with a generative AI model ⁵.

The price of encoding a misunderstanding

Robodebt, settlement approved 2026. Between 2015 and 2019 the Australian government raised welfare debts by averaging annual tax office income across fortnights, although the social security law assesses income in the fortnight it was received ⁶. The Royal Commission found in 2023 that the scheme was neither fair nor legal ⁶. In June 2026 the Federal Court approved an additional A$548.5 million settlement, including A$475 million in compensation for about 125,000 registered claimants, on top of an earlier settlement reached in 2021 ⁷. About 450,000 people are estimated to have been affected ⁷.

The Post Office Horizon inquiry, 2025. Volume 1 of the inquiry's final report, published in July 2025, describes hundreds of postmasters, managers and assistants wrongly accused of theft and fraud because of faults in the Horizon system ⁸. Among the questions the inquiry examined was how the Post Office handled discrepancies that postmasters reported ⁸. Some people were held liable for small sums the system said were missing, and others were wrongly imprisoned ⁸. The discrepancy reports were the one place where the gap between the system and the branch showed up, and the process treated them as losses to recover.

Klarna, 2025. In February 2024 Klarna said its AI assistant was handling 75% of customer chats ⁹. In May 2025 its chief executive told Bloomberg the AI-focused path had not been the right one and began recruiting human agents again, so that customers could always reach a person ⁹. Klarna has not published which kinds of cases went wrong. The 75% figure measured how many chats the assistant took, which says nothing about how it handled the ones that needed judgment.

The industry pattern

The failure rate for automation programs has been high for a decade. EY reported in 2016 that as many as 30 to 50% of initial robotic process automation (RPA) projects fail, based on the projects it was called in to rescue ¹⁰. The figure is old and comes from a firm that sells the remedy, so treat it as an indication of scale. Deloitte's 2022 survey of 479 executives in 35 countries found process fragmentation was the top barrier to scaling automation in each of its previous four surveys, and 52% cited an inability to change processes or ways of working ¹¹. Only 23% of those organizations were using process mining, the discipline of reconstructing processes from system logs, although 82% agreed it produced better outcomes ¹¹.

AI agents are inheriting the same problem with a larger budget. Gartner predicted in June 2025 that more than 40% of agentic AI projects will be canceled by the end of 2027, citing rising costs, unclear business value and inadequate risk controls ¹².

Why the Usual Approach Fails

The documented process is a minority of the real one

Process mining research shows how far real work drifts from its description. The 2019 Business Process Intelligence Challenge released the purchase-to-pay event log of a multinational paints and coatings company: 251,734 purchase order items, 1.6 million events and 42 activities, with every item assigned to one of four documented matching flows ¹³. One published analysis of that log found 14,333 distinct paths through those four flows ¹⁴.

The labels did not describe the work either. In the flow defined as "invoice before goods receipt," 91% of cases actually had the goods receipt first or at the same moment ¹⁴. In 21% of those cases staff still performed a payment-block removal step the case did not need, which raised the median time from order to payment from 72 to 91 days ¹⁴. A bot built from the flow definitions would have automated the wrong sequence for nine cases in ten.

Exceptions are where the cost lives

This is an old finding. In Michael Hammer's 1990 account of Ford's accounts payable department, more than 500 people spent most of their time on mismatches between purchase orders, receiving documents and invoices ¹⁵. Ford's first plan, new computer systems on the existing process, aimed for a 20% headcount cut. Redesigning the process so that mismatches could not arise produced a 75% reduction and cut the items matched from 14 to three ¹⁵. The source is 36 years old. The same economics apply to a bot or an agent today, because the routine path is usually the cheapest part of a workflow already.

Three habits keep the gap hidden

Business cases are written from the policy, the vendor demo or a workshop with managers, all of which describe work as imagined. The clerk who keeps a spreadsheet of vendors whose invoices always need correcting is rarely interviewed.

Rules and code drift apart without an owner. The Targeted Compliance Framework failed at exactly this joint: a law changed, and nobody was responsible for checking that the system changed with it ²³.

Success is measured as hours removed from the happy path. Few business cases count the exceptions pushed back to staff, the rework they create or the wrong decisions made at volume. The NIST AI Risk Management Framework asks organizations to understand and document the intended purposes, context-specific laws and deployment settings of an AI system, and to define processes for human oversight ¹⁶. Most organizations cannot meet that expectation for a workflow they have never measured.

The Framework: The Work-as-Done Baseline

Before funding, require a baseline built from four kinds of evidence. Each catches what the others miss.

flowchart LR
    A[Interview<br/>front-line staff<br/>and supervisors] --> E[Measured baseline<br/>paths, exceptions,<br/>rework, rules]
    B[Observe<br/>real cases<br/>at the desk] --> E
    C[Mine event logs<br/>timestamps from<br/>source systems] --> E
    D[Trace rules<br/>policy and law<br/>to code] --> E
    E --> F[Fund, redesign<br/>or stop]

Interview. Talk to the people who do the work and the people who handle its failures, separately from their managers. Ask what they do when the system is wrong, which cases they set aside and what they keep outside the system.

Observe. Sit with staff for a sample of real cases, including the slow ones. Observation finds the copying between systems and the phone calls that no log records.

Mine the logs. Most ERP, case management and ticketing systems record a timestamp for every step. Process mining reconstructs every path cases actually took and compares them with the documented model, a technique called conformance checking ¹⁷. This gives volume and distribution, which interviews cannot.

Trace the rules. For any workflow that applies policy, law or contract terms, list each rule and the place in the system that implements it, with a named owner. This is the check the Targeted Compliance Framework never had ².

The baseline ends in one of three decisions. Fund automation of the paths that are stable and high-volume. Redesign first where exceptions dominate, as Ford did. Stop when the process depends on judgment the organization cannot yet specify.

The numbers to ask for

The same seven numbers become the acceptance test after go-live. Falling cycle time with a rising exception rate means cost moved to another team.

Costs, Risks, and Tradeoffs

How to Act

Questions to Ask Your Team

1. For the workflow we are about to automate, what share of last quarter's cases followed the documented process from start to finish?
2. What is the 90th percentile cycle time, and what happens to the cases that take that long?
3. Who on the front line was interviewed for this business case, and what workarounds did they describe?
4. Which spreadsheets or side systems does this process depend on today, and what happens to them after go-live?
5. Which rules from policy, law or contract does this automation apply, and who updates the system when one of them changes?
6. Where do exceptions go after automation, and has that team's workload been costed?
7. What exception and rework rates will tell us, six months after launch, that this project failed?

Closing

Automation repeats the process it was given, faster and at greater volume. When that process is the version in the manual, the automation repeats the manual's errors and drops the judgment that staff were quietly supplying. Require a measured baseline of the work as done before you fund automation or AI on any workflow. Measure exceptions, rework and the rules that govern each decision, assign an owner for every gap, and judge the project afterward against the same numbers.