Run them on every change

An eval you ran once is a screenshot. An eval that runs every time the prompt, the model or the code changes is a safety net.

The rule that makes it grow: every bug you fix becomes a test case. The Sunday mistake from lesson 2 gets a case the day you fix it, and it stays in the suite for good. If a later change brings the bug back, the suite says so before a customer does.

Keep two lists:

• Regression set: things the bot already does right. These should pass nearly every time. A failure means something broke.

• Capability set: things you want it to do but it doesn't yet. Most fail today, and that's fine. It's your to-do list, and you watch the pass rate climb.

Run the regression set on every change. Keep it small and mostly code-graded, so it's fast and costs pennies.

When a case fails, read the reply before you blame the bot. Sometimes the bot was wrong. Sometimes the check was too strict and rejected a good answer. Either way, you've learned something.

Here's the shape of a regression run in CI: run every case, count passes, and fail the build if anything that used to pass doesn't.

import { cases } from "./cases"; // [{ name, input, check }]
import { answer } from "./bot";  // your prompt, wrapped in one Claude call

let failed = 0;
for (const c of cases) {
  const ok = c.check(await answer(c.input));
  if (!ok) failed++;
  console.log(ok ? "✓" : "✗", c.name);
}

console.log(`${cases.length - failed}/${cases.length} passed`);
process.exit(failed === 0 ? 0 : 1); // a failure stops the deploy

That's the whole idea in a dozen lines: inputs, a task that produces replies, and checks that score them. Matt Pocock's AI Hero guide calls the same three parts data, task and scorers. When you outgrow a hand-rolled loop, an eval runner such as his Evalite adds the report, the history and the LLM scorers for you.

One more habit from Anthropic: test both sides. When you fix the Sunday bug, a lazy fix is "say we're closed" to every opening-hours question. A case that only checks Sunday would pass it. Pair it with a case where the bot should not say closed.

Your turn. Get Claude to write that balanced pair for the Sunday fix.