Strip the vocabulary and evaluation is four familiar practices. A golden set: twenty to a hundred real cases with agreed-correct outputs, run after every prompt, model, or tool change, exactly like regression tests. Sampled review: a human grades a fixed percentage of live outputs weekly against a short rubric. Tripwires: two or three metrics that page a human when they move (error rate, customer escalations, spend per task). Version discipline: every change to the agent is logged, so a quality dip has a diff to blame.
If your company can run a QA function for human work, it already knows how to do this. The novelty is only that the worker is stochastic, which makes the golden set non-negotiable rather than optional.
Public model benchmarks measure general capability on public tasks. Your workflow is neither. A model that tops a leaderboard can be mediocre at your specific extraction task, and a cheaper model with a better-built harness can beat it. The only benchmark that matters is your golden set, run on your tasks, with your context.
This also answers "which models are actually good?": build the fifty-case golden set once, and every new model release becomes an afternoon of testing instead of a debate.
Diff the surface (1 min). Check tripwires (1 min). Read the plan, not the transcript (2 min). Spot-check three cases: routine, edge, high-stakes (3 min). Verify machine-checked invariants (1 min). Review the checks, not the work.
When an agent produces a day of work, a human cannot re-do it to verify it; review must be structural. Five moves, roughly eight minutes. Diff the surface (one minute): what was created, changed, touched; anything outside expected scope is an instant flag. Check the tripwires (one minute): spend, error count, anomalous tool calls. Read the plan, not the transcript (two minutes): review the agent's stated approach and decisions, the way you would skim a report's summary. Spot-check three items (three minutes): one routine case, one edge case, one high-stakes case, graded against the rubric. Verify the invariants (one minute): the things that must always hold (nothing sent externally, tests pass, totals reconcile), ideally checked by machine and merely confirmed by you.
The ratio holds because you are not checking the work; you are checking the checks. That is what management has always been.
Traditional user acceptance testing assumes deterministic behavior: same input, same output. AI features break that assumption, so extend UAT with three additions: a golden set with tolerance bands instead of exact matches (accuracy above a threshold, not equality), adversarial cases (ambiguous inputs, prompt injection attempts, out-of-scope requests), and a monitored soft-launch phase, because sampled production behavior is part of acceptance, not an afterthought. Sign-off criteria become statistical: 95 percent of golden cases within tolerance, zero critical invariant violations, tripwires quiet for two weeks.
Against a golden set of your own real cases, plus weekly sampled human review with a rubric, plus tripwire metrics on error rate, escalations, and spend. Public benchmarks do not predict performance on your specific workflow.
Structurally: diff what was touched, check tripwire metrics, read the agent's plan, spot-check three representative cases, and confirm machine-verified invariants. Review the checks rather than redoing the work.
Add a golden set with tolerance bands, adversarial test cases, and a monitored soft launch. Acceptance criteria become statistical thresholds plus zero critical invariant violations, not exact-match assertions.
Bring one AI question from this article. We will tell you what we would do, whether or not you hire us.
Book a 15-Min Chat →