Evaluations Are the Test Suite for AI

The Decision in Front of You

Your organization probably runs several AI features already: a support assistant, a document summarizer, an agent that drafts code or answers policy questions. Each one passed some form of review before launch. The decision now is whether you can prove, on any given day, that each one still behaves the way it did when it was approved.

Ordinary software gives you that proof through automated tests. A change that breaks a test does not ship. AI systems need the same discipline, with a difference that matters to the budget: their behavior can change without anyone on your team touching the code. A model vendor retrains, a colleague edits a system prompt, a policy document in the knowledge base gets replaced. Each of these can change answers your customers rely on.

The stakes are legal and financial as well as technical. In February 2024 a Canadian tribunal held Air Canada liable for a refund policy its website chatbot described incorrectly, and rejected the airline's argument that the chatbot was a separate entity responsible for its own statements ⁶. In October 2025 Deloitte agreed to repay the final installment of an AU$440,000 government contract after a report prepared partly with a GPT-4o-based tool chain was found to contain fabricated citations and an invented quote from a court judgment ⁵. An academic reader found the errors after publication ⁵.

What's Happening

The model vendors keep finding regressions after release

OpenAI, April 2025. On April 24 and 25, OpenAI shipped an update to GPT-4o in ChatGPT that made the model noticeably more sycophantic, meaning overly eager to agree with and please the user ¹. OpenAI began rolling back to the previous version on April 28 ¹. Its postmortem says the offline evaluations looked good and an A/B test showed users liked the new model. Some expert testers said it felt slightly off, and the company launched anyway ¹. No deployment evaluation tracked sycophancy specifically ¹. OpenAI committed to treat behavior problems such as hallucination and deception as launch-blocking even when A/B metrics look good, and to improve its offline evaluations ¹.

Anthropic, August and September 2025. Three separate infrastructure bugs degraded Claude's responses for several weeks ³. One routed some requests to servers configured for a different context window and affected 16% of Claude Sonnet 4 requests in the worst hour, on August 31 ³. Another, a server misconfiguration, corrupted output for Claude Opus and Sonnet models during late August and early September ³. Each bug showed different symptoms on different platforms, so user reports did not point to a single cause ³. Anthropic said it would run more sensitive evaluations in more places ³.

Anthropic, March and April 2026. Users of Claude Code reported weeks of declining quality before Anthropic published a postmortem on April 23, 2026 ². The causes were three product changes: a lower default reasoning setting, a caching bug that discarded the model's working context after an hour of idle time, and a system prompt that capped response length ². The model weights had not changed ². Anthropic's coding evaluations caught a 3% quality drop from the prompt change; the article does not say how the other two causes were found ². Anthropic reset usage limits for all subscribers and committed to broader evaluations for every system prompt change ².

These companies build the models. Their regressions came from a feedback signal, server configuration, caching, and prompt wording, and none of them required new model weights. A company building on their models carries the same exposure in its own prompts and data, plus whatever the vendor changes underneath it.

Most teams evaluate less than they monitor

LangChain's State of Agent Engineering survey collected 1,340 responses in late 2025 ⁴. 57% of respondents had agents in production, and about a third named quality as the main barrier to getting there ⁴. Yet only 52% ran offline evaluations against test sets, and 37% evaluated production traffic ⁴. Observability was far more common: 89% had some form of it ⁴. The sponsor sells evaluation and tracing tools, so treat the figures as indicative.

Tracing records what an agent did. Judging whether that was correct, and whether it was more or less correct than last week, takes an evaluation.

Why the Usual Approach Fails

Most AI features reach production after informal review: a few people try a few prompts and decide the results look right. OpenAI's guidance lists judging quality by impression as an anti-pattern ¹¹. That review happens once, at launch, and leaves nothing that can be rerun.

Manual review cannot keep pace with change. A system built on a hosted model can see its behavior shift with each vendor release, each prompt edit, and each document added to retrieval. Nobody repeats a manual review for each of those events, so most of them ship untested.

A/B tests and satisfaction scores measure the wrong thing. OpenAI's own users preferred the sycophantic model in testing ¹. People reward answers that agree with them. A quality bar has to measure correctness and policy compliance directly.

Public benchmarks describe the model in general. A leaderboard score says little about whether it applies your refund policy correctly, and OpenAI's guidance also warns against generic metrics, unrepresentative datasets, and evaluating only after release ¹¹.

Standards already expect more. The NIST AI Risk Management Framework, NIST AI 100-1, makes Measure one of its four core functions ⁷. Its subcategories call for documented test sets and metrics, for measuring performance under conditions like deployment, for showing the deployed system is valid and reliable, and for monitoring behavior once the system is in production ⁸. ISO/IEC 42001:2023, the AI management system standard, requires organizations to decide what to monitor and measure and by what method, to keep evidence of results, and to reassess AI risk when a significant change is proposed ⁹. An auditor working from either document will ask for test results tied to each change.

The Framework: The Change Gate

An evaluation, or eval, is an automated test for an AI system: a set of inputs, the expected behavior for each, and a grader that scores the output. Graders can be code that checks an exact answer, a second model that scores against a rubric, or a person ¹⁰. Anthropic's guidance distinguishes capability evals, which measure how well the system handles hard tasks, from regression evals, which should pass close to 100% of the time and exist to catch backsliding ¹⁰.

The Change Gate treats every change that can alter behavior the same way, whether your team made it or a vendor did.

flowchart LR
    A[Change proposed<br/>model, prompt, data,<br/>tool, setting] --> B[CI eval run<br/>regression suite<br/>behavior checks]
    B --> C[Release gate<br/>thresholds met<br/>owner signs off]
    C --> D[Canary<br/>small share of traffic<br/>scored live]
    D --> E[Production<br/>sampled scoring<br/>human review]
    E --> F[New failure<br/>becomes a test case]
    F --> B

Build the suite from reality. Start with 20 to 50 cases drawn from actual failures and actual user requests, which is enough to detect the large effects early changes produce ¹⁰. Add the questions that carry legal or financial weight: refunds, eligibility, pricing, safety. Each case needs an expected outcome a reviewer would agree with.

Run it in CI. The suite runs automatically on every pull request that touches a prompt, a model setting, a tool definition, or retrieval configuration. Automating the scoring is what makes it cheap enough to run every time ¹¹.

Gate releases on thresholds. Agree on the pass rate for each suite before the release, and write it down. Regression suites should sit near 100% ¹⁰. Behavior failures such as fabricated facts, policy violations, or unsafe advice block the release outright, which is the rule OpenAI adopted after its rollback ¹.

Score production. Run the same graders on a sample of live traffic, and send a share of low-scoring cases to a person for review. Anthropic's guidance pairs automated evals with production monitoring, user feedback, and transcript review, since no single method catches every issue ¹⁰.

Calibrate the judges. Model-based graders drift and disagree with people. Check them against human ratings on a fixed sample, and recalibrate when agreement falls ¹⁰¹¹.

A hypothetical case

Consider a mid-size insurer with a customer service assistant that answers coverage questions from policy documents. The vendor ships a new model version, and the team adopts it after a day of informal testing because responses read better.

Three weeks later a complaint shows that the assistant has been telling some customers that water damage from a burst pipe is covered under a policy that excludes it. Nobody can say how many customers heard that answer, or when it started.

With a Change Gate, the model upgrade runs a regression suite that includes the twenty coverage questions most often escalated to agents. The exclusion question fails, the release is blocked, and the team adjusts the retrieval prompt and reruns the suite before switching models. The Air Canada ruling suggests a tribunal would hold the insurer to its assistant's answer ⁶. The eval record is how the insurer shows it took reasonable care.

Costs, Risks, and Tradeoffs

How to Act

Questions to Ask Your Team

1. Which AI features do we run in production, and which model version does each one call today?
2. If our vendor changed that model tonight, how would we know by tomorrow?
3. For our highest-risk AI feature, how many automated test cases exist, and when did they last run?
4. What pass rate does a prompt change need before it ships, and who decided that number?
5. Which customer complaints about AI answers in the last quarter are now test cases?
6. What share of production AI responses do we score, and who looks at the low scores?
7. If a regulator or a court asked how we checked an answer our assistant gave, what would we show?

Closing

No CTO would ship a payment system without automated tests and then let a vendor swap out its libraries without warning. Many AI features run exactly that way today. Treat evaluations as the test suite for AI: build them from real cases, run them on every model, prompt, and data change, block releases that fail, and keep scoring the system in production so the next vendor update is caught by your pipeline before a customer finds it.