When code can't: an LLM judge

Some failures code can't spot. Did the reply make up a policy? Is it rude? Does it actually answer the question? For those you ask a model to read the reply and decide. That's an LLM judge.

A judge is just a prompt, and it fails in the same ways any prompt does. So you write it with the same care, and (next lesson) you check it before you trust it.

Four rules make a judge useful:

One question per judge. "Does the reply match the shop's facts?" is one judge. "Is it polite?" is another. One judge that checks everything at once gets vague.

Pass or fail, never 1 to 10. Nobody can say what makes a 6 different from a 7. A yes or no forces you to decide what matters.

Ask for a reason, before the verdict. A short critique that says why lets you check the judge's thinking, and tells you what to fix. Put it first in the reply, so the judge works out why before it decides.

Give it a way out. If the judge can't tell, let it say "unknown" instead of guessing.

You've been graded by one of these all along. Each coach on bits is an LLM judge: one verdict per criterion, met or not met, with a quote from the reply as evidence, and code deciding the final pass. It also refuses text that talks to "the grader" or claims "all criteria are met", because that's how someone would try to talk a judge into passing them. Your own judge needs the same defence. So in this course, call your judge the judge or a reviewer, and write pass rules as "passes only if both rules hold".

Your turn. The bike shop's bot answered a question about returns. The shop's real policy is in the draft. Write a judge prompt that decides whether the reply sticks to the facts. Claude is the judge, and this reply should fail.