Evals for agents

An agent doesn't give one reply. It takes steps: it calls a tool, reads the result, calls another, then answers. So "did the reply look right?" isn't enough. An agent can say "You're booked for Friday!" and have booked Thursday.

Anthropic's guide gives you two things to grade. The outcome: what's actually true at the end, like the booking in the database. The transcript: every step the agent took to get there.

Two layers, per the FAQ:

1. End to end. Treat the agent as a black box. Define precise success on the final state (the database row, the file, the API side effect), not on the final message.

2. Step-level diagnostics. When it fails, score the trajectory: tool choice, argument extraction, error handling, context retention. Assert tool name, arguments, result and resulting state separately, in code where you can.

Anthropic adds: grade outcomes rather than exact paths (rigid path checks fail valid strategies), give partial credit on multi-part tasks, and start each trial from a clean, isolated environment so shared state doesn't make results flaky. Across many failures, the FAQ's transition failure matrix (last good state against first failure) shows where workflows tend to break.

There's a second twist: agents vary from run to run. Anthropic uses two measures:

• pass@k: did at least one of k tries succeed? Fine when a person picks the best attempt.

• pass^k: did all k tries succeed? This is what a customer feels, because every booking has to work.

If one try succeeds 90% of the time, three tries in a row all succeed only about 73% of the time (0.9 × 0.9 × 0.9).

Your turn. Below is a made-up transcript where the agent told the customer the right thing and booked the wrong day. Write a judge that grades the outcome, then finds the first step that went wrong: the first one that used a wrong value, even if it didn't change anything yet.