Check your judge against a person

Your judge failed the bad reply. Good. Would it also pass a good one? Or does it fail everything, which would look identical on that single test?

An LLM judge is a prompt, so it has the same problems as the prompt it's grading. Shankar and colleagues put it bluntly: model graders inherit the flaws of the models they grade, so a person has to check them. You check a judge the way you check anything else: against answers you already know.

The answers you already know come from a person who knows what good looks like: the shop owner, a support lead, you. They label real replies pass or fail, each with a reason. Then you run your judge on the same replies and count how often it agrees.

There's a surprise waiting. Shankar's research found that people change their minds about the rules while labelling. You don't fully know what "good" means until you've looked at real replies. So label first, then write the judge's rules from what you learned.

Watch out for one trap. Say a person labelled 20 replies: 18 good, 2 bad. Your judge says "pass" to all 20. That's 18 out of 20, 90% agreement, and the judge is useless: it missed both failures.

So count two things separately:

• Failures caught. Of the replies the person failed, how many did the judge fail?

• Good replies passed. Of the replies the person passed, how many did the judge pass?

A judge that never says "fail" scores perfectly on the second and zero on the first. Husain calls these the true positive and true negative rates, and reports them separately because overall agreement hides this.

Your turn: a tiny calibration run. The shop owner labelled two bot replies: A passes, B fails. Write a judge prompt that grades both, and see if it agrees with the owner on both.

There's no AI coach here, on purpose. Agreement with a person's labels is arithmetic, so code checks it: your judge must say pass for A and fail for B.