Test your prompt · Prompting that works

You've written a prompt, run it, and the reply looks great. Is the prompt done? Not yet. You've seen one answer to one input. The next input might be the one that breaks it.

The fix is small and it's the habit that separates people who build with Claude from people who demo it: keep a few test cases, and re-run them every time you change the prompt. Anthropic calls these evaluations, or evals, and puts them before prompt engineering, not after.

Start by writing down what “good” means, in a way you can check. “Labels support messages well” can't be checked. “Replies with exactly one of billing, bug or other, and gets all ten test messages right” can.

Then pick test cases the way a sceptic would. Include a few normal inputs, and then the awkward ones Anthropic suggests: an empty or irrelevant input, a very long one, and an ambiguous one where even people might disagree. Real messages from your users beat anything you invent.

Let code do the grading when it can. Anthropic's order of preference is code first (an exact match, or checking that a phrase is there), then Claude grading with a clear rubric, and people only when you must. More cases graded automatically beats a few graded by hand.

You've been using one all course: the checks under each exercise are a tiny code-graded eval. They're regex patterns run on your prompt, on the server.

Here's a whole eval in a few lines. It runs four test messages through a labelling prompt and counts exact matches. The output panel replays an example run: it's recorded output, not a live call to Claude.

import { label } from "./label"; // your prompt, wrapped in one Claude call

const cases = [

{ input: "I was charged twice this month", expect: "billing" },

{ input: "The app crashes when I upload a photo", expect: "bug" },

{ input: "", expect: "other" },

{ input: "My card was charged twice and now the app won't open", expect: "billing" },

];

let passed = 0;

for (const { input, expect } of cases) {

const got = (await label(input)).trim().toLowerCase();

const ok = got === expect;

if (ok) passed++;

console.log(ok ? "✓" : "✗", JSON.stringify(input), "→", got);

}

console.log(`${passed}/${cases.length} passed`);

TypeScriptUTF-8● lines 3–8: highlighted

OUTPUT

npx tsx eval.ts

✓ "I was charged twice this month" → billing

✓ "The app crashes when I upload a photo" → bug

✓ "" → other

✗ "My card was charged twice and now the app won't open" → bug

3/4 passed

example output, not a live run

The last case is ambiguous: it's a billing problem and a bug. The prompt never said what to do when a message fits two labels, so Claude made a reasonable call that didn't match yours. That's not Claude being wrong. It's a gap in the prompt that only a test could find.

Change one thing at a time, then re-run every case. A fix for one case can quietly break another, and you'll only see it if you run them all.

Your turn. Write the labelling prompt and fix the gap: list the three labels, ask for only the label, add a rule for messages that fit two labels, and test it on the tricky message from the eval.

Your turnWrite the labeller, with a rule for mixed messages, and run the tricky case

Run your prompt and the replies land here.

Down the rabbit hole