I shipped a prompt change once that made my support agent 4% better on the metric I was watching and 30% worse at actually closing tickets. The eval was green. The users were furious. That's when I learned most agent evals measure the wrong thing — they measure whether the output sounds right, not whether the agent did the right thing.
If you're building agents and your evals are a handful of "does the response look good" checks scored by another model, you don't have evals. You have vibes with a number attached. Here's how I write evals for AI agents that actually catch regressions.
Assert on behavior, not prose
The single biggest upgrade: stop grading the final text and start grading what the agent did. Did it call the right tools, in a reasonable order, with the right arguments? Did it modify the files it was supposed to and leave the rest alone? Those are checkable facts, not opinions.
For a coding agent, my eval doesn't read the summary. It checks: does the test suite pass after the run, did the agent touch only the files in scope, did it avoid editing the lockfile. Three boolean assertions. No LLM judge anywhere. They never flake and they catch the failures that matter.
assert run_tests() == 0, "tests fail after agent run"
assert touched_files <= allowed_files, f"out of scope: {touched_files - allowed_files}"
assert "package-lock.json" not in touched_files
That last one came from a real incident. An agent kept "helpfully" regenerating the lockfile and breaking CI. One assertion, never happened again.
Test the trajectory, not just the endpoint
Agents fail in the middle. The final answer can be correct by luck after a disaster of a path — six redundant searches, a tool call with garbage args that happened to error gracefully, a wrong turn it recovered from. If you only check the endpoint, you're blind to a trajectory that's one small change away from failing outright.
So I log every event and assert on the path. Did it call search before answering a question that needed fresh info? Did it make more than two redundant tool calls? Did it ever hit is_error: true on a tool result? You can score a trajectory without a judge — most of it is counting and pattern-matching over the event stream.
The model behavior here shifted recently, by the way. The newer Opus models reach for tools more conservatively. A search-first assertion that passed on an older model can start failing not because the agent got worse but because it now answers from context when it shouldn't. That's exactly the kind of regression a trajectory eval catches and an output eval misses.
When you do need an LLM judge, pin it
Some things genuinely need judgment — was the tone right, did the summary capture the key point, was the explanation actually correct. Fine. Use an LLM judge. But treat the judge like production code, because it is.
Three rules I never break. One: pin the judge model and version. If your judge silently upgrades, your scores shift and you'll chase a regression that's actually a judge change. Two: give the judge a concrete rubric, not "rate this 1-10." "Does the response cite a specific file and line number? yes/no" beats a vibe score every time. Three: spot-check the judge against human labels on a sample. A judge you've never audited is a random number generator with good PR.
Here's a trap specific to the current models. If you tell a judge "only flag high-severity issues" or "be conservative," the newer models follow that literally — they find the bugs, then decline to report the ones below your bar. Your measured recall drops and it looks like the agent regressed when really the judge just got obedient. Tell the judge to report everything with a confidence and severity, and filter downstream. Move the filtering out of the judging step.
Build the eval set from your failures
Don't sit down and brainstorm test cases. You'll write the ones you already handle. Instead, every time an agent screws up in prod, that exact scenario becomes an eval case. The lockfile incident, the flip-flop loop, the time it answered from stale context — each one is now a permanent regression test.
This is the part people skip because it's not glamorous. But an eval set grown from real incidents is worth ten brainstormed ones. It's load-bearing in a way synthetic cases never are, because every case is a bug that actually bit you.
Run them on every prompt change
The whole point is the loop. You change a prompt, you run the evals, you see the trajectory assertions before you ship. My support-agent disaster happened because I eyeballed a few outputs and shipped on a hunch. If I'd had trajectory evals — "did it actually resolve or just sound resolved" — the 30% drop would've been red before it hit a single user.
Evals aren't a quality gate you bolt on at the end. They're the only thing standing between "I think this prompt is better" and "this prompt is better." One of those is engineering. The other is a guess wearing a lab coat.
Grow them from your scars. Assert on what the agent did. Pin your judge. And never, ever ship a prompt change on vibes again.