Measure it · Retrieval without tears

Desk

Verified 3 Nov 2026agents 0.33.0wrangler 4.157.0vitest 5.0.5@cloudflare/vitest-plugin 1.5.1

Start from e-07 on GitHub. Starting here, or behind? Get the code at that tag:

Shell

git clone https://github.com/ryanyogan/klippy
cd klippy
git checkout e-07

You've stacked up chunking, vectors, keywords, RRF, context and a reranker. Did each one help? Without a number, you're guessing. This lesson adds the number, and a bug worth learning from.

Don't run this lesson's eval. At tag e-08, pnpm eval:retrieval writes to and deletes from your real klippy-notes index, and can destroy your real vectors. Read along, find the bug in step 4, and run the fixed version in lesson 9.

1. recall@k

Take questions whose answers you know. For each, run the search and look at the top k results. recall@k is the share of questions whose answer is among them: 1.00 means every answer was found.

For Klippy, k = 5, because the reranker hands the model five passages. If the answer isn't in those five, the model can't cite it, however good it is.

2. A question set

apps/agent/test/eval/corpus.ts holds six made-up documents and twelve questions, each with the document that answers it:

export const questions: { q: string; answer: string }[] = [

{ q: 'What vegetable does my dog like?', answer: 'pip.md' },

{ q: 'When are the vaccinations due for the dog?', answer: 'pip.md' },

{ q: 'What is the flight number to Portugal?', answer: 'lisbon-trip.md' },

{ q: 'Which tram should we take early?', answer: 'lisbon-trip.md' },

...

{ q: 'When is the quarterly review?', answer: 'work-notes.md' },

{ q: 'How do we undo a bad release?', answer: 'work-notes.md' },

TypeScriptUTF-8at e-08● lines 2–5: apps/agent/test/eval/corpus.ts

Good questions are worded the way a person asks, not copied from the document: "vegetable" for carrots, "Portugal" for Lisbon. Questions that reuse the document's words test nothing but keyword search. Claude is good at drafting them. Try it:

Your turnGet Claude to draft retrieval test questions

Goal Get Claude to write a handful of questions a real customer might ask that this passage answers, in their words rather than the passage's

What good looks likeThe coach checks the reply

Asks for questionsnot checked yet
Sets how manynot checked yet
Includes the passagenot checked yet
Ran at 1.0not checked yet
Writes questionsnot checked yet
Questions sound like a customer, not reworded lines from the passagenot checked yet
Every question is about something the passage coversnice to have
Questions cover different facts: how long, what's covered, and the deadlinenice to have

Run your prompt and the replies land here.

Read every drafted question before it goes in the set. A question the document doesn't answer measures nothing.

3. The scorer

apps/agent/test/eval/recall.ts is the whole formula:

// recall@k: the share of questions whose answer is among the top k results. 1 means every answer was found.

export function recallAt(k: number, results: string[][], answers: string[]) {

const found = results.filter((titles, i) => titles.slice(0, k).includes(answers[i])).length;

return found / answers.length;

}

TypeScriptUTF-8at e-08● lines 2–5: apps/agent/test/eval/recall.ts

It's pure, so test/recall.spec.ts checks the arithmetic in the normal test run. Copy corpus.ts and create recall.ts now: neither one touches an index, and both stay the same at e-09.

4. The eval, and its bug

test/eval/retrieval.eval.ts ingests the corpus under a fresh owner, runs every question four ways (keywords, vectors, hybrid, hybrid + rerank), and prints recall@5 for each. It needs the real Workers AI and Vectorize, so vitest.eval.config.mts turns on remoteBindings, and it runs only with pnpm eval:retrieval, never in pnpm test or CI. Here's how it sets up and cleans up at e-08:

beforeAll(async () => {

await applyD1Migrations(env.DB, env.TEST_MIGRATIONS);

for (const [title, body] of Object.entries(documents)) {

const source = await queueSource(env.DB, { owner, kind: 'file', title, body });

await saveChunks(env.DB, source!, chunk(body));

const { results } = await env.DB.prepare('SELECT id, owner, source_id, context, text FROM chunks WHERE source_id = ?').bind(source!.id).all<ChunkRow>();

await upsertChunks(env, results);

ids.push(...results.map((row) => `chunk-${row.id}`));

...

afterAll(async () => {

await env.VECTORIZE.deleteByIds(ids);

TypeScriptUTF-8at e-08● lines 7–8: apps/agent/test/eval/retrieval.eval.ts

Spot it? upsertChunks(env, …) writes to env.VECTORIZE, your real index, under ids chunk-1, chunk-2, … The eval's local D1 starts empty, so its chunk ids start at 1, the same ids your real chunks have. The upsert overwrites your real vectors with the corpus, and the cleanup deletes them. A fresh owner doesn't help: ids aren't scoped by owner.

The lesson in general: a measurement that writes gets its own resources. Lesson 9 gives it a separate -eval index, ids of its own, and a guard that refuses to start otherwise.

5. What it measured

We ran it once, on a recording account whose index held only test vectors. The numbers:

recorded output

pnpm eval:retrieval --reporter=verbose

keywords only 1.00

vectors only 1.00

hybrid 1.00

hybrid + rerank 1.00

hybrid + rerank, recall@1 0.92

ok What vegetable does my dog like? -> pip.md

ok When are the vaccinations due for the dog? -> pip.md

ok What is the flight number to Portugal? -> lisbon-trip.md

ok Which tram should we take early? -> lisbon-trip.md

ok What pressure for the tyres? -> bike.md

ok How often should I oil the chain? -> bike.md

ok How much salt goes in the bread? -> sourdough.md

ok What temperature do I bake the loaf at? -> sourdough.md

ok When is the quarterly review? -> work-notes.md

MISS How do we undo a bad release? -> bike.md

ok When can the tomatoes be planted outside? -> garden.md

ok How do I get rid of slugs? -> garden.md

Read it honestly: this set is too easy. Every method finds every answer in its top five, so recall@5 can't tell them apart. Only recall@1 shows a miss: "undo a bad release" should find the rollback line in work-notes.md, and the reranker put bike.md first.

So the next move isn't tuning. It's a harder set: your own saved things, the questions you really ask, paraphrases, exact codes like TP1337, near-duplicate notes. Make it hard enough that the methods differ, then change one thing at a time and watch the number.

6. Prove it

Copy test/recall.spec.ts and run pnpm test: ours was 21 files, 95 tests passed. Don't copy retrieval.eval.ts, vitest.eval.config.mts or the eval:retrieval script from this tag: lesson 9 gives you the fixed ones.

Your code matches tag e-08, minus its eval files (changes from e-07). Don't run its eval: lesson 9 fixes it.