How Do You Know Your AI Agent Is Actually Good? Evals for Operators

Evals are QA, not rocket science

Strip away the vocabulary and evaluation is four practices you already know.

A golden set: twenty to a hundred real cases with agreed-correct outputs, run after every prompt, model, or tool change, exactly like regression tests. Sampled review: a human grades a fixed percentage of live outputs each week against a short rubric.

Tripwires: two or three metrics that page a human when they move, such as error rate, customer escalations, and spend per task. Version discipline: every change to the agent is logged, so a dip in quality has a diff to blame.

If your company can run a QA function for human work, it already knows how to do this. The only new thing is that the worker gives a slightly different answer each time. That makes the golden set non-negotiable instead of optional.

Benchmarks tell you almost nothing

Public model benchmarks measure general capability on public tasks. Your workflow is neither.

A model that tops a leaderboard can be mediocre at your specific extraction task, and a cheaper model inside a better-built harness can beat it. The only benchmark that matters is your golden set, run on your tasks, with your context.

This also settles the "which models are actually good?" argument for good. Build the fifty-case golden set once, and every new model release becomes an afternoon of testing instead of a debate.

The 8-Minute Check: reviewing 8 hours of agent work

When an agent turns out a full day of work, no human can redo it to verify it. So the review has to be structural: five moves, roughly eight minutes.

Diff the surface (one minute): what was created, changed, touched. Anything outside expected scope is an instant flag. Check the tripwires (one minute): spend, error count, odd tool calls.

Read the plan, not the transcript (two minutes): review the agent's stated approach and decisions, the way you would skim the summary of a report. Spot-check three items (three minutes): one routine case, one edge case, one high-stakes case, graded against the rubric.

Verify the invariants (one minute): the things that must always hold, such as nothing sent externally, tests passing, totals reconciling. Ideally a machine checks them and you only confirm.

Eight minutes for eight hours works because you are not checking the work. You are checking the checks. That is what management has always been.

UAT when the software includes AI

Traditional user acceptance testing assumes the same input gives the same output. AI features break that assumption, so you extend UAT with three additions.

First, a golden set with tolerance bands instead of exact matches: accuracy above a threshold, not equality. Second, adversarial cases: ambiguous inputs, prompt injection attempts, out-of-scope requests. Third, a monitored soft-launch phase, because sampled production behavior is part of acceptance, not an afterthought.

Sign-off becomes statistical: 95 percent of golden cases within tolerance, zero critical invariant violations, tripwires quiet for two weeks. When those three hold, you can ship it and sleep.

Repeat Testers: what teams with reliable agents do differently

Some teams ship agents that just work, week after week, and from the outside it looks like luck.

It is not luck. They have three habits the rest skip, and each habit was paid for by somebody else's public mistake. We call these teams Repeat Testers.

They test the same thing many times. An agent that gets a task right once has told you very little. Researchers behind a benchmark called tau-bench ran a leading model through realistic customer service tasks and found it finished fewer than half of them.

Then they asked the harder question: does it get the same task right eight times in a row? In the retail tests, the answer was yes less than 25 percent of the time. Your customer does not meet the average run.

They meet whichever run they get.

Anthropic's guide to agent evals does the arithmetic. An agent that succeeds 75 percent of the time per try passes three tries in a row only about 42 percent of the time. Run the golden set once and you are reading the flattering number.

They grade the company, not the chatbot. In November 2022 a man named Jake Moffatt asked Air Canada's website chatbot about bereavement fares. It told him he could claim the discount after flying.

He could not.

He took the airline to a tribunal, and Air Canada argued the chatbot was responsible for its own words. The tribunal called that "a remarkable submission" and ordered the airline to pay $812.02 in Canadian dollars. The lesson cost Air Canada almost nothing in cash and a great deal in headlines: whatever your agent says, you said.

They keep two piles of tests. One pile is things the agent cannot do yet, which gives the team a hill to climb. The other is things it already does, and that pile should pass nearly every time.

Mix the piles and you cannot tell progress from breakage.

"This sounds like an engineering project."

You may be thinking this is fine for a company with an AI team and too much for yours. Look at who got caught.

Air Canada's bot was a help widget. New York City's MyCity chatbot was a help widget too, built to answer small-business questions, and five months after launch reporters found it telling owners to do things that break the law.

Neither one was a moonshot. Both were the kind of simple assistant a 30-person company turns on in an afternoon. And neither failure needed an engineer to catch it.

It needed somebody to ask the bot twenty real questions and read the answers.

That is the whole job at small scale. Twenty real questions, the answers you would accept, a rerun after every change, and a rerun several times over for anything a customer will see. A spreadsheet is enough to start.

Once that habit exists, the harness is where you wire it in so it runs without you. Until it exists, no tool will save you.

What to read next, and why

Demystifying evals for AI agents (guide): The clearest plain-language walk through agent testing we know. Read the part on running the same test several times.

Moffatt v. Air Canada, as reported by CBC (news): Five minutes, and you will never again think of a chatbot's answer as somebody else's problem.

The tau-bench paper (research): You only need the summary. It is the best evidence that "it worked when I tried it" is not a test.

The AI Incident Database (database): A public list of what went wrong for other people. Search your own use case before you launch.

So the simple takeaway is this: an agent is as good as its worst ordinary day, and the only way to see that day before your customer does is to run the same test more than once.