Tutor:
Time to put it together. You'll write a judge, run it on replies a person has already labelled, and see if it agrees with them on every one. That's lessons 4 and 5 in one go.
Tutor:
The pipeline, end to end:
Error analysis builds the failure taxonomy; each category becomes tasks with checkable success criteria and, ideally, a reference solution.
Code graders cover everything objective; constrain output (schemas, enums) so they can.
LLM judges cover judgement calls: one criterion each, binary with a critique, an escape hatch, and the ground truth in the prompt.
Calibration against a principal expert's labels, with a held-out test set, reporting true positive and true negative rates rather than raw agreement.
Regression gate in CI (deterministic, a rate across trials), capability suite on a schedule, judges sampled.
Agents: outcome on final state, step-level diagnostics on failure, pass^k for anything a user relies on.
Tutor:
The task: the bike shop uses Claude to summarise its weekly meeting. The owner wrote one rule for a good summary: every action item names who owns it, and it states only decisions the meeting actually made. Then they labelled three summaries: A passes, B fails, C fails.
Write a judge that applies that rule to all three. Two things grade you:
Code checks your judge agrees with the owner on A, B and C.
The coach, an LLM judge itself, reads your judge's notes and checks they name the real problem in B and in C, and that your prompt spells out the pass rule.
Your turn
Goal Get A pass, B fail and C fail, with notes that name what's wrong in B and in C
What good looks likeThe coach checks the reply
- Includes the meeting notesnot checked yet
- Includes all three summariesnot checked yet
- Asks for pass or failnot checked yet
- Asks for JSONnot checked yet
- Ran at 1.0not checked yet
- Parses as JSON with A, B, C and notesnot checked yet
- Agrees with the owner on A: passnot checked yet
- Agrees with the owner on B: failnot checked yet
- Agrees with the owner on C: failnot checked yet
- The notes say B doesn't name who orders the tyre stocknot checked yet
- The notes say C claims Sunday opening was decided, when the meeting chose to waitnot checked yet
- The prompt spells out the pass rule: owners on every action item, and only decisions actually madenot checked yet
- The prompt asks for a reason with each verdictnice to have
Run your prompt and the replies land here.
Tutor:
Three labelled replies is a smoke test, not a validated judge. Before you trust a judge like this in production, label a hundred or more real summaries, keep some back as a test set, and check how many failures it catches and how many good summaries it passes. Then put it in your regression run.
Finish this checkpoint and the course badge is yours.