Start from real failures

Where do good test cases come from? Not from brainstorming in an empty file. They come from things that actually went wrong.

Anthropic's advice for a first eval suite is 20 to 50 simple tasks drawn from real failures: the checks you already do by hand, what users commonly ask, your bug tracker and your support inbox. You don't need hundreds to start.

The method is borrowed from qualitative research. Husain and Shankar call it the most important activity in evals, because it decides which evals you write.

1. Dataset. Representative traces of real use, or realistic synthetic ones before launch.

2. Open coding. Read traces and write free-text notes, focusing on the first failure in each. Do at least 30 yourself.

3. Axial coding. Group the notes into a failure taxonomy and count each category.

4. Iterate until saturation. Keep sampling until new traces stop adding or changing categories. A pool of about 100 diverse traces is their guardrail. An agent can help find more examples of a failure you've already named, but don't let it name your failures for you.

Then turn categories into tasks. Each task needs an input, success criteria you can check, and ideally a reference solution: a known good output that passes, which proves the task is solvable and the grader works.

A test case is only as good as its "expect". Compare:

• "Replies helpfully about opening hours." Nobody can grade that the same way twice.

• "Says the shop is closed on Sundays." Code can check it.

One more rule from Anthropic: test both sides. If the bot should say "closed" on Sunday, also test that it doesn't say "closed" on Saturday. A suite with only one side rewards a bot that says "closed" to everyone.

Your turn. The shop owner read last week's chats and wrote three notes. Get Claude to turn them into test cases you could run.