A very good next-word guesser

Here's the whole trick, in one sentence. A language model looks at the text so far and guesses what comes next.

The unit is a token, not a word (lesson 3). One forward pass takes the token sequence so far and returns one score (a logit) per entry in the vocabulary, on the order of a hundred thousand of them. Softmax turns the scores into a distribution. One token is sampled, appended, and the pass runs again. That is autoregressive decoding, and it ends when the model emits an end-of-turn token or hits max_tokens.

So a reply is never computed as a whole. It is a sequence of draws, each one conditioned on everything before it, your prompt included. Step through one below; the probabilities are illustrative, the shape is real.

Some steps are near-deterministic (“of” after “sound”), some are close (“roof” 0.52 against “window” 0.38). Claude samples rather than taking the argmax, so the 0.38 branch wins about 4 runs in 10 and the same prompt gives different replies. Two practical consequences. The current Messages API exposes no sampling parameters, so you can't turn this off; design for variance instead (lesson 8 has the habits). And Anthropic notes that even a fixed setting is not fully deterministic across calls, so never build on byte-identical output.

“Just autocomplete” undersells the objective. Minimising next-token error over every kind of text forces the model to encode grammar, facts, register and argument structure, because each of those lowers the error. And the one-token-at-a-time output does not mean one-token-at-a-time planning: Anthropic's interpretability work found Claude choosing a rhyme word before writing the line that leads to it. The decoding is sequential; the computation behind each step is not that naive.

Your turn. Ask Claude to finish a sentence with just one word, then run it a few times. Does the word change?