Skip to content
bits
Sign inStart free

Evals that matter

How do you know your prompt works? Not from one good reply. This course turns "it looked fine when I tried it" into a habit you can trust: test cases from real failures, code graders first, an LLM judge you've checked against a person, and a regression set that runs on every change. Every exercise runs live on Claude, and most are graded the way the lesson teaches.

  • Intermediate
  • 8 lessons
  • 59 min
  • text + code
Start the course →

Lesson 1 · One good reply proves nothing · 6 min

Syllabus

M1: Why evals

  1. 1. One good reply proves nothing — not started6 min · +30 neurons
  2. 2. Start from real failures — not startedPrompt exercise7 min · +70 neurons

M2: Grading

  1. 3. Let code grade first — not startedPrompt exercise7 min · +60 neurons
  2. 4. When code can't: an LLM judge — not startedPrompt exercise8 min · +60 neurons
  3. 5. Check your judge against a person — not startedPrompt exercise8 min · +70 neurons

M3: Keep it working

  1. 6. Run them on every change — not startedPrompt exerciseCode exercise7 min · +60 neurons
  2. 7. Evals for agents — not startedPrompt exercise8 min · +80 neurons
  3. 8. Checkpoint: a judge you can trust — not startedPrompt exerciseProject8 min · +80 neurons

On finishing

Evals that matter badgePlus 510 neurons and a certificate

Neurons are the points you earn for each lesson you finish.

Your progress stays in this browser. Sign in to keep it on every device.

What you'll learn

  1. Turn real failures into test cases with clear, checkable expectations.
  2. Grade with code first, and design output so code can grade it.
  3. Write an LLM judge that gives pass or fail with a reason, and check it against a person's labels.
  4. Keep a regression set running on every change, and grade agents on outcomes as well as steps.

How you'll learn

8
lessons, every one a chat with Claude
7
prompt exercises
1
code exercise
1
checkpoint project

Source

This course is 9 markdown files on GitHub. Spot a typo? Fix it in one pull request.

content/courses/evals-that-matter/ ↗

Written by The bits team · updated