Chunking · Retrieval without tears

Retrieval works on passages, not whole documents. Before you can search, you cut every document into pieces called chunks. A search returns the best chunks, and only those go to Claude.

Why not search whole documents? A 40-page manual is about everything at once, so it matches every question a little and none of them well. And if it wins, you're back to pasting 40 pages. Anthropic's guide puts typical chunks at a few hundred tokens. Embedding models can take much more (Voyage's current models take 32,000 tokens per input), so the limit isn't the model. It's how precise you want each match to be.

Where you cut matters as much as how big. Here's the same warranty page split two ways:

Every 120 characters vs On paragraphsillustrativecompare

promptOrbit Bikes warranty page, split into chunks

Every 120 characters

Chunk 1: “E-bike batteries have a 2-year warranty from delivery. It covers capacity below 70%. Claims must be fil” Chunk 2: “ed within 30 days of noticing the fault. Frames have a lifetime warranty for the first owne”

On paragraphs

Chunk 1: “E-bike batteries have a 2-year warranty from delivery. It covers capacity below 70%. Claims must be filed within 30 days of noticing the fault.” Chunk 2: “Frames have a lifetime warranty for the first owner.”

Fixed-size cuts slice through words and mix topics. Cutting on the document's own structure (headings, then paragraphs) keeps each idea in one piece.

Three habits that pay off. Cut on structure, and only pack small paragraphs together until you hit a size limit. Overlap a little: repeat the last paragraph at the start of the next chunk, so an idea that spans a boundary survives in at least one of them. Keep metadata: store the source, title and heading with every chunk. You'll need them to cite sources in lesson 7.

Here's a paragraph chunker with one paragraph of overlap. It measures size in characters, which is close enough for chunking:

export type Chunk = { id: string; source: string; text: string };

// Pack paragraphs into chunks of up to maxChars, repeating the last

// paragraph of a multi-paragraph chunk at the start of the next one.

export function chunk(source: string, text: string, maxChars = 1500): Chunk[] {

const paragraphs = text.split(/\n\s*\n/).map((p) => p.trim()).filter(Boolean);

const chunks: Chunk[] = [];

let current: string[] = [];

for (const p of paragraphs) {

const size = current.join("\n\n").length + p.length;

if (current.length > 0 && size > maxChars) {

chunks.push({ id: `${source}#${chunks.length}`, source, text: current.join("\n\n") });

current = current.length > 1 ? [current[current.length - 1]!] : [];

}

current.push(p);

}

if (current.length > 0) {

chunks.push({ id: `${source}#${chunks.length}`, source, text: current.join("\n\n") });

}

return chunks;

}

TypeScriptUTF-8● lines 10–17: the packing loop

Example output from running it on a short warranty page (yours will differ):

OUTPUT

npx tsx try-chunk.ts warranty.md

warranty.md#0 412 chars · E-bike batteries have a 2-year warranty…

warranty.md#1 655 chars · Claims must be filed within 30 days…

warranty.md#2 388 chars · Frames have a lifetime warranty…

example output, not a live run

Notice that a chunk can still lose its meaning on its own. Lesson 5 fixes that.

Down the rabbit hole