Last lesson: everything the model can use is in one long text. This lesson: how it reads that text. To guess the next token well, the model has to know what the earlier ones mean here. “Bank” by a river isn't “bank” with a vault. And “it” could be almost anything.
The mechanism is attention, the core of the transformer architecture (Vaswani et al., 2017) that Claude and nearly every current LLM are built on.
Each token's vector produces a query, a key and a value. A token's query is scored against every earlier token's key; softmax turns the scores into weights; the token's representation is updated with the weighted sum of those tokens' values. A causal mask hides later positions, so each token only attends backwards, which is what makes left-to-right generation work. The result is that “it” can carry the meaning of “animal” into its own position before the next-token prediction is made.
Below is one head on one sentence, with made-up weights chosen to show a clean pattern.
“it” looks most at “animal” (62%), “street” (18%), “The” (6.0%).
“It” leans hard on “animal”, so the model treats “it” as the animal. Swap “tired” for “wide” and a good model's “it” would lean towards “street” instead.
Real models run many heads per layer in parallel, each with its own query/key/value projections, so different heads can specialise (coreference, syntax, which delimiter is open), and stack dozens of layers, each attending over the previous layer's output. The clean single-head picture above is a simplification of a very wide, very deep version of the same operation.
This is the mechanism that reads the context window. Every token in the window is a key and a value for every later token, so the work grows with the length of the context, and the keys and values for the whole prefix have to exist before the first output token. Two engineering consequences. Long prompts cost more and have higher time-to-first-token. And a prefix that doesn't change between calls (system prompt, documents, tool definitions) should be prompt-cached: on Claude, cache reads cost 10% of the input price, and the cached prefix's computation is reused rather than redone.
Your turn. Write a sentence where a word like “it” could point two ways, and ask Claude what it refers to. Then swap one word and run it again.
Run your prompt and the replies land here.