Last lesson: everything the model can use is in one long text. This lesson: how it reads that text. To guess the next token well, the model has to know what the earlier ones mean here. “Bank” by a river isn't “bank” with a vault. And “it” could be almost anything.
The part of the model that sorts this out is called attention. It's the big idea inside the transformer, the design almost every modern LLM uses, Claude included.
Here's the intuition. At each token, the model asks: which earlier tokens matter for me? It gives each one a weight, then blends in information from the ones with the most weight. Only earlier tokens, never later ones: it's writing left to right.
Click a word below to see what it looks back at. These weights are illustrative, made up to show one clean pattern.
“it” looks most at “animal” (62%), “street” (18%), “The” (6.0%).
“It” leans hard on “animal”, so the model treats “it” as the animal. Swap “tired” for “wide” and a good model's “it” would lean towards “street” instead.
The real thing is messier. A model runs many attention heads side by side, each free to track something different (who did what, what a word refers to, which bracket is still open). Then it stacks many of these layers, so later layers build on what earlier ones worked out.
This is where the context window gets used. Attention is how the model reads across everything in it, your system prompt and old turns included. Every new token looks back over all the earlier ones, so the longer the text, the more looking there is to do. That's one reason long prompts cost more and run slower.
Your turn. Write a sentence where a word like “it” could point two ways, and ask Claude what it refers to. Then swap one word and run it again.
Run your prompt and the replies land here.