Run them on every change

An eval you ran once is a screenshot. An eval that runs every time the prompt, the model or the code changes is a safety net.

The rule that makes it grow: every bug you fix becomes a test case. The Sunday mistake from lesson 2 gets a case the day you fix it, and it stays in the suite for good. If a later change brings the bug back, the suite says so before a customer does.

Anthropic separates capability evals (start at a low pass rate, a hill to climb) from regression evals (should sit near 100%, so a dip means backsliding). Promote a capability case to the regression set once it passes reliably.

For CI, the FAQ recommends a small, purpose-built set: core features, a regression case for every past bug, and the known edge cases. Favour deterministic checks there because they're cheap. Run LLM judges on a schedule or a sample.

Non-determinism. The same prompt can pass and then fail. Run important cases several times and gate on a rate (for example 3 of 3, or 9 of 10), not on one lucky trial.

Read transcripts on failure. Anthropic's point: a failure is either a real mistake or a grader rejecting a valid answer. Both need fixing, in different places.

Saturation. A suite where everything always passes has stopped telling you anything new. Add harder cases, or run it less often.

Here's the shape of a regression run in CI: run every case, count passes, and fail the build if anything that used to pass doesn't.

import { cases } from "./cases"; // [{ name, input, check }]
import { answer } from "./bot";  // your prompt, wrapped in one Claude call

let failed = 0;
for (const c of cases) {
  const ok = c.check(await answer(c.input));
  if (!ok) failed++;
  console.log(ok ? "✓" : "✗", c.name);
}

console.log(`${cases.length - failed}/${cases.length} passed`);
process.exit(failed === 0 ? 0 : 1); // a failure stops the deploy

That's the whole idea in a dozen lines: inputs, a task that produces replies, and checks that score them. Matt Pocock's AI Hero guide calls the same three parts data, task and scorers. When you outgrow a hand-rolled loop, an eval runner such as his Evalite adds the report, the history and the LLM scorers for you.

One more habit from Anthropic: test both sides. When you fix the Sunday bug, a lazy fix is "say we're closed" to every opening-hours question. A case that only checks Sunday would pass it. Pair it with a case where the bot should not say closed.

Your turn. Get Claude to write that balanced pair for the Sunday fix.