One good reply proves nothing

You tweak a prompt. You try it on the message you've been testing with. The reply is better. You ship it.

A week later a customer sends a message you never tried, and the "better" prompt gets it wrong. Nothing warned you, because the only test was your eyes on one reply.

That's the habit this course replaces. An eval is a fixed set of test inputs, each with a way to decide pass or fail, that you run every time you change something. It's a unit test for a prompt.

The failure mode is sampling bias plus non-determinism. You evaluate on the one input you remember, once, and a model's output varies between runs. Here's a prompt change that wins on the tested input and regresses on an untested one.

Version 2 fixed Sundays and quietly stopped answering in Spanish. If you'd kept both messages as test cases and re-run them after the change, you'd have seen it in seconds.

The vocabulary, as Anthropic's agent-evals guide defines it:

• Task (test case): defined inputs plus success criteria.

• Trial: one attempt. Outputs vary between runs, so you run several and report a rate, not a single result.

• Grader: logic that scores some aspect of the result: code, a model, or a person.

• Transcript: the full record of a trial (outputs, tool calls, intermediate results). You read these when a case fails.

• Eval suite: a set of tasks measuring one capability or behaviour.

Two kinds of suite matter early. Capability evals start with a low pass rate: they're the hill you're climbing. Regression evals should pass at close to 100%: they catch you sliding back.

Avoid generic scores such as "helpfulness: 4.2". Husain and Shankar call them false confidence. They measure something abstract, not the failures your users actually hit.

The rest of the course builds this up one piece at a time: where good test cases come from, how code grades them, when to bring in a model as judge, how to check that judge, and how to keep it all running.