Where do good test cases come from? Not from brainstorming in an empty file. They come from things that actually went wrong.
Anthropic's advice for a first eval suite is 20 to 50 simple tasks drawn from real failures: the checks you already do by hand, what users commonly ask, your bug tracker and your support inbox. You don't need hundreds to start.
The habit has a name, error analysis, and it's mostly reading.
Collect real conversations your bot had (or, before launch, realistic ones).
Read them yourself. For each, write a short note about the first thing that went wrong.
Group the notes into kinds of failure and count them.
Fix the biggest group first, and turn each kind of failure into test cases.
Husain and Shankar suggest reading at least 30 conversations yourself before asking any tool to help, and to keep going until new ones stop showing you new kinds of failure.
A test case is only as good as its "expect". Compare:
"Replies helpfully about opening hours." Nobody can grade that the same way twice.
"Says the shop is closed on Sundays." Code can check it.
One more rule from Anthropic: test both sides. If the bot should say "closed" on Sunday, also test that it doesn't say "closed" on Saturday. A suite with only one side rewards a bot that says "closed" to everyone.
Your turn. The shop owner read last week's chats and wrote three notes. Get Claude to turn them into test cases you could run.
Goal Get one test case per failure, each with a real customer message and an expectation specific enough to check
Run your prompt and the replies land here.