An agent doesn't give one reply. It takes steps: it calls a tool, reads the result, calls another, then answers. So "did the reply look right?" isn't enough. An agent can say "You're booked for Friday!" and have booked Thursday.
Anthropic's guide gives you two things to grade. The outcome: what's actually true at the end, like the booking in the database. The transcript: every step the agent took to get there.
Grade the outcome first. Did the right thing happen? Check the booking that exists, not the agent's message about it.
Don't insist on one exact path. If the agent checks availability twice before booking, that's fine as long as the result is right. Anthropic warns that grading the exact sequence of steps makes evals rigid and fails good agents.
When the outcome is wrong, read the transcript to find the first step that went wrong. That's what you fix. Check each tool call on its own: was it the right tool, with the right inputs, and did the agent use the result correctly?
AskThe customer: book my bike in for a service this Friday.
step 1 of 4
There's a second twist: agents vary from run to run. Anthropic uses two measures:
pass@k: did at least one of k tries succeed? Fine when a person picks the best attempt.
pass^k: did all k tries succeed? This is what a customer feels, because every booking has to work.
If one try succeeds 90% of the time, three tries in a row all succeed only about 73% of the time (0.9 × 0.9 × 0.9).
Your turn. Below is a made-up transcript where the agent told the customer the right thing and booked the wrong day. Write a judge that grades the outcome, then finds the first step that went wrong: the first one that used a wrong value, even if it didn't change anything yet.
Goal Get a fail outcome, the number of the first step that used a wrong value, and a reason that names the wrong date
Run your prompt and the replies land here.