Let code grade first

You have test cases. Now something has to decide pass or fail for each one, every time, without you reading replies by hand.

Anthropic's order of preference is clear: code first. A few lines of code are fast, free, give the same answer every time, and are easy to debug. Only reach for a model as the grader when code really can't tell.

Code can check more than you'd think:

• Exact match. The reply is exactly billing.

• Contains. The reply mentions "£15".

• Format. The reply is valid JSON with the fields you asked for.

• Length. The reply is under 50 words.

• Pattern. The reply has a date in it.

The catch: code is literal. If the right answer is "fifteen pounds" and your check looks for "£15", a correct reply fails. So the trick is to ask Claude for output code can check, like JSON with a fixed set of values.

Here's what a code grader looks like. Each check is a plain function that takes the reply and returns pass or fail:

type Check = (reply: string) => boolean;

const isJson: Check = (reply) => {
  try { JSON.parse(reply); return true; } catch { return false; }
};

const hasIntent = (want: string): Check => (reply) =>
  isJson(reply) && JSON.parse(reply).intent === want;

const underWords = (n: number): Check => (reply) =>
  reply.trim().split(/\s+/).length <= n;

Notice hasIntent only works because the reply is JSON with an intent field, and it checks isJson first so a prose reply fails instead of crashing the run. If the bot answered in a friendly paragraph, you'd need a model to read it. Designing the output is half of designing the eval.

Your turn. The bike shop wants to sort incoming messages so bookings, cancellations and questions go to the right place. Write the prompt so Claude replies in a shape code can grade, then run it on a cancellation.

This exercise has no AI coach. Code grades it, exactly as this lesson describes: does the reply parse, does it have the fields, and are the values right.