Tutor:
Time to put it together. You'll write a judge, run it on replies a person has already labelled, and see if it agrees with them on every one. That's lessons 4 and 5 in one go.
Tutor:
Quick recap of the course, as a checklist you can reuse:
Test cases come from real failures, with expectations you can check.
Code grades first. Ask for output code can read, like JSON.
A judge answers one question, pass or fail, with a reason first, and can say "unknown".
Check the judge against a person's labels. Count failures caught and good replies passed separately.
Run the regression set on every change, and test both sides of a fix.
For agents, grade the outcome first, then find the first bad step.
Tutor:
The task: the bike shop uses Claude to summarise its weekly meeting. The owner wrote one rule for a good summary: every action item names who owns it, and it states only decisions the meeting actually made. Then they labelled three summaries: A passes, B fails, C fails.
Write a judge that applies that rule to all three. Two things grade you:
Code checks your judge agrees with the owner on A, B and C.
The coach, an LLM judge itself, reads your judge's notes and checks they name the real problem in B and in C, and that your prompt spells out the pass rule.
Your turn
Goal Get A pass, B fail and C fail, with notes that name what's wrong in B and in C
What good looks likeThe coach checks the reply
- Includes the meeting notesnot checked yet
- Includes all three summariesnot checked yet
- Asks for pass or failnot checked yet
- Asks for JSONnot checked yet
- Ran at 1.0not checked yet
- Parses as JSON with A, B, C and notesnot checked yet
- Agrees with the owner on A: passnot checked yet
- Agrees with the owner on B: failnot checked yet
- Agrees with the owner on C: failnot checked yet
- The notes say B doesn't name who orders the tyre stocknot checked yet
- The notes say C claims Sunday opening was decided, when the meeting chose to waitnot checked yet
- The prompt spells out the pass rule: owners on every action item, and only decisions actually madenot checked yet
- The prompt asks for a reason with each verdictnice to have
Run your prompt and the replies land here.
Tutor:
Three labelled replies is a smoke test, not a validated judge. Before you trust a judge like this in production, label a hundred or more real summaries, keep some back as a test set, and check how many failures it catches and how many good summaries it passes. Then put it in your regression run.
Finish this checkpoint and the course badge is yours.