You've now got a lot of dials: chunk size, overlap, embedding model, BM25 or not, contextual chunks, reranking, how many results to keep. Without a number, you're tuning by vibes.
The good news: you can measure retrieval on its own, before Claude writes a word. If the right chunk never reaches the prompt, no prompt can save the answer.
The measure is recall@k. Write down a set of real questions, and for each one, the chunk that answers it. Run your search. Recall@k is the share of questions where that chunk shows up in the top k results.
Twenty questions, right chunk in the top 5 for seventeen of them: recall@5 is 17 ÷ 20 = 0.85. Anthropic's contextual retrieval tests used exactly this, at k = 20, and reported the misses (1 − recall@20).
Run it once per setup, and keep the misses: reading why a question missed teaches you more than the score. Here's the kind of table you're after (illustrative numbers):
A few tips. Twenty to fifty questions is plenty to start, as long as they're real: pull them from support tickets or search logs. Phrase them like users do, not by copying the chunk's wording, or keyword search will look better than it is. And check answers separately: high recall with bad answers means the problem is in the prompt, not the search.
No test questions yet? Claude can draft them. Your turn: ask for five questions a customer might ask that this passage answers, phrased the way customers talk, not copied from the passage.
Run your prompt and the replies land here.