Let code grade first

You have test cases. Now something has to decide pass or fail for each one, every time, without you reading replies by hand.

Anthropic's order of preference is clear: code first. A few lines of code are fast, free, give the same answer every time, and are easy to debug. Only reach for a model as the grader when code really can't tell.

Code graders cover exact match, substring and regex checks, schema validation, length limits, and execution tests (does the generated SQL run, do the tests pass). They're objective and reproducible, so a failing case means something.

Their weakness is brittleness. Anthropic notes they reject valid variations that don't match the expected pattern. The fix is to constrain the output so the valid space is small: an enum of labels, a JSON schema, a fixed date format. Structured outputs make the schema part a guarantee rather than a request. The checks below then read fields, not prose.

Here's what a code grader looks like. Each check is a plain function that takes the reply and returns pass or fail:

type Check = (reply: string) => boolean;

const isJson: Check = (reply) => {
  try { JSON.parse(reply); return true; } catch { return false; }
};

const hasIntent = (want: string): Check => (reply) =>
  isJson(reply) && JSON.parse(reply).intent === want;

const underWords = (n: number): Check => (reply) =>
  reply.trim().split(/\s+/).length <= n;

Notice hasIntent only works because the reply is JSON with an intent field, and it checks isJson first so a prose reply fails instead of crashing the run. If the bot answered in a friendly paragraph, you'd need a model to read it. Designing the output is half of designing the eval.

Your turn. The bike shop wants to sort incoming messages so bookings, cancellations and questions go to the right place. Write the prompt so Claude replies in a shape code can grade, then run it on a cancellation.

This exercise has no AI coach. Code grades it, exactly as this lesson describes: does the reply parse, does it have the fields, and are the values right.