Check your judge against a person

Your judge failed the bad reply. Good. Would it also pass a good one? Or does it fail everything, which would look identical on that single test?

An LLM judge is a prompt, so it has the same problems as the prompt it's grading. Shankar and colleagues put it bluntly: model graders inherit the flaws of the models they grade, so a person has to check them. You check a judge the way you check anything else: against answers you already know.

The calibration loop:

1. One principal expert labels. Husain recommends one person (two at most) whose judgement defines the standard. Binary label plus a critique.

2. Label enough to measure. The FAQ suggests 100 to 200 examples for each failure mode, split about 10–20% as few-shot examples in the judge prompt, 40–45% as a dev set you iterate against, and 40–45% held out for one final test.

3. Iterate the judge prompt against the dev set until agreement is high. Husain uses over 90% as a working bar.

4. Test once on the held-out set, so you're not fooling yourself with a prompt tuned to the examples it's graded on.

Expect criteria drift (Shankar et al.): grading outputs changes the criteria. Some criteria only appear once you've seen specific outputs, so writing the rubric before looking at any outputs doesn't work well.

Watch out for one trap. Say a person labelled 20 replies: 18 good, 2 bad. Your judge says "pass" to all 20. That's 18 out of 20, 90% agreement, and the judge is useless: it missed both failures.

So count two things separately:

• Failures caught. Of the replies the person failed, how many did the judge fail?

• Good replies passed. Of the replies the person passed, how many did the judge pass?

A judge that never says "fail" scores perfectly on the second and zero on the first. Husain calls these the true positive and true negative rates, and reports them separately because overall agreement hides this.

Your turn: a tiny calibration run. The shop owner labelled two bot replies: A passes, B fails. Write a judge prompt that grades both, and see if it agrees with the owner on both.

There's no AI coach here, on purpose. Agreement with a person's labels is arithmetic, so code checks it: your judge must say pass for A and fail for B.