You tweak a prompt. You try it on the message you've been testing with. The reply is better. You ship it.
A week later a customer sends a message you never tried, and the "better" prompt gets it wrong. Nothing warned you, because the only test was your eyes on one reply.
That's the habit this course replaces. An eval is a fixed set of test inputs, each with a way to decide pass or fail, that you run every time you change something. It's a unit test for a prompt.
Here's the trap in one picture. A bike shop's support bot got a prompt change. Version 2 fixed the message the owner kept testing. It also broke one nobody tried.
promptDo you open on Sundays?
Yes, we're open every day!
Sorry, we're closed on Sundays and Mondays. We're open Tuesday to Saturday, 9 to 6.
prompt¿Arregláis pinchazos?
Sí, arreglamos pinchazos por 15 libras.
Yes, we fix punctures for £15.
Version 2 fixed Sundays and quietly stopped answering in Spanish. If you'd kept both messages as test cases and re-run them after the change, you'd have seen it in seconds.
A few words you'll see all course. They come from Anthropic's guide to evals:
A test case (or task) is one input plus what a good reply must do.
A trial is one run of it. Claude can answer differently each time, so important cases get run more than once.
A grader decides pass or fail. It can be code, another model, or a person.
An eval suite is your set of test cases, run together.
The important part is "what a good reply must do". "Be helpful" can't be checked. "Says we're closed on Sundays" can.
The rest of the course builds this up one piece at a time: where good test cases come from, how code grades them, when to bring in a model as judge, how to check that judge, and how to keep it all running.