Evals as contracts: how we regression-test agent behavior

The upgrade nobody would sign off on

Last spring a logistics client of ours had an agent system reconciling carrier invoices against their TMS - about 11,000 invoices a month, with a human review gate on anything over a two percent variance. A new model generation came out that was faster, cheaper, and measurably better at the exact document-extraction work this agent did. The client's ops leadership refused the upgrade for five months.

Their reasoning was completely sound. The system had been certified through a painful six-week validation the previous year. Nobody could say what the new model would do differently, so the only honest answer was to re-run the whole certification. Six weeks of validation to capture maybe eight percent cost savings is a bad trade, so the answer was no, indefinitely.

This is the quiet failure mode of enterprise agent deployments. Not hallucinations, not runaway actions - ossification. Systems freeze on the model they launched with because change is unverifiable, and within eighteen months the client is running production workflows on a deprecated model with a sunset date. We decided the certification itself was the thing to fix.

What we mean by a contract

An API contract does not promise that the implementation is unchanged. It promises that observable behavior at the boundary is unchanged. Consumers do not re-audit a library's internals on every release; they check the contract, and the contract is checked by machines.

We apply the same logic to agents. The eval suite is not a research artifact or a launch-time quality gate. It is the binding, versioned definition of what this agent is certified to do. The model, the prompts, the scaffolding - all of that is implementation. The contract is the suite, and the rule we sign with clients is simple: any change that passes the contract ships without re-certification, and any change that fails it does not ship at all.

This reframing changes who owns the suite. It is not owned by whoever built the agent. It is co-owned with the client's process owners, the same people who used to sign the certification documents. Their sign-off moves from certifying a system once to certifying a contract that then certifies every future version automatically. That is the whole trick, and it is the reason a model swap becomes a Tuesday instead of a quarter.

Contracts come from the audit, not the incident log

When we start an engagement, the first month is a workflow audit - we sit with the ops team and map what actually happens: the ERP screens, the exception spreadsheet nobody admits to, the tribal rules like never auto-approve a credit memo from this one supplier because their pricing file is chronically stale. Most firms use that audit to design the agent. We also use it to design the contract, because every rule someone tells us during an audit is a test case wearing a trench coat.

A revenue ops contract we run for a mid-market software client currently holds 340 cases. Roughly the composition: about 60 percent are golden-path replays, real historical transactions with known-correct outcomes, sampled across every segment the audit identified. About 25 percent are edge cases harvested from the exception spreadsheet - the partial refunds, the currency mismatches, the PO that references a cancelled contract. The last 15 percent are behavioral invariants, which are the most important and the least obvious: the agent must escalate rather than guess when a customer record is ambiguous, must never write to the CRM before the ERP transaction commits, must produce a citation to a source document for every dollar figure it asserts.

The rule of thumb we give clients: if a behavior matters enough to put in a runbook, it matters enough to put in a contract. Anything a reviewer would reject at the human gate is, by definition, a missing test.

Three tiers, and only one of them blocks

Early on we made the classic mistake of treating every eval equally, which meant a model upgrade that improved 300 cases and cosmetically changed the phrasing on 4 would fail the run and die in review. Contracts need severity tiers, exactly like API versioning distinguishes breaking from non-breaking changes.

Tier one is invariants, and they are absolute: zero tolerance, any failure blocks the release, no human override in the pipeline. These are things like never execute a payment action above the delegated threshold, always route flagged entities to the review gate, never fabricate a reference number. A candidate model that passes 339 of 340 cases but fails one invariant does not ship. This tier is small - usually 30 to 60 cases - and it is the part clients read line by line before signing.

Tier two is accuracy against golden outcomes, measured as a rate with an explicit floor, typically the trailing production baseline minus one percent. It is allowed to trade: a new model can miss cases the old one passed as long as the aggregate stays above the floor and none of the new misses are in a segment flagged as sensitive during the audit. Segment-level floors matter more than the headline number - we have seen a model improve overall accuracy while quietly getting worse on exactly the invoice format used by the client's largest supplier.

Tier three is style and workflow ergonomics: summary formats, tone of escalation notes, ordering of evidence. Failures here never block. They open tickets. Conflating tier three with tier one is how eval suites lose credibility with engineers, and once engineers stop trusting the suite, you are back to vibes-based certification.

If a behavior matters enough to put in a runbook, it matters enough to put in a contract.

A contract that only runs at upgrade time is a dead contract

The suite runs three ways. First, in CI on every change to prompts, tools, or orchestration code - full tier-one and tier-two runs, about 20 minutes, blocking merge. Second, as the upgrade gate: a candidate model runs the full contract plus a shadow week, executing against live traffic in parallel with the incumbent while only the incumbent's actions take effect, with disagreements diffed nightly. Third, and this is the part most teams skip, a ten percent sample of the contract replays against production continuously, because vendors update hosted models under stable names and drift does not schedule an appointment.

The human review gates feed the contract in return. Every case a reviewer overrides at the gate becomes a candidate test case, triaged weekly with the client's process owner. This is where the contract earns its keep long-term: the suite grows in exact proportion to the ways reality has surprised the system. Our finance ops contracts grow by roughly 15 to 25 cases a month in the first half year, then settle to a trickle. When the trickle stops entirely, that is not success - that usually means the review gate has gotten lazy, and we go audit the gate.

One discipline point: contract changes are versioned and signed like the original. Loosening a tier-two floor requires the same client sign-off as the initial certification. A contract you can quietly edit under pressure is not a contract, it is a suggestion.

What it buys, and where it honestly stops

The logistics client from the opening now upgrades on a different cadence. Their last model swap went: candidate announced on a Monday, full contract run that afternoon, one tier-two segment regression found in handwritten delivery exceptions, a targeted prompt fix, clean run Wednesday, shadow week, promoted the following Thursday. Ten days, about six hours of human attention, most of it reading diffs. The certification that used to cost six weeks now costs a coffee's worth of compute, and they have upgraded three times in twelve months.

Honesty requires the limits. Contracts verify behavior you thought to specify, and a genuinely novel model failure mode can sit in the untested space - this is why the shadow week and the review gates stay even when the suite is green, and why we never contract-certify an expansion of the agent's authority, only implementation swaps within existing authority. New capabilities go through the month-by-month rollout like anything else.

But that boundary is the point. An eval contract does not promise the agent is perfect. It promises the agent has not silently become a different agent. In enterprise ops, where the alternative is freezing on aging models or re-certifying the world every quarter, that narrower promise turns out to be the one worth engineering for.