A very good next-word guesser

Here's the whole trick, in one sentence. A language model looks at the text so far and guesses what comes next.

Not the whole answer at once. One small piece, called a token (often a word, sometimes part of one). Then it adds that piece to the text and guesses again. And again. Until it guesses that it's done.

Slow it right down and one reply is a loop of five moves. Read everything so far, your prompt included. Give every token it knows a score, and it knows tens of thousands. Turn the scores into odds that add up to 100%. Draw one. Add it to the text, and go back to the start. A reply is that loop running until the model draws the token that means “I'm done”.

Step through a reply below and watch the odds at each draw. The numbers are illustrative, but the shape is real.

Notice that some steps are nearly certain (“of” after “sound”) and some are close calls (“roof” or “window”). The model doesn't always take the top pick. Claude samples from the odds, so a 38% option wins roughly 4 times in 10. That's why the same prompt can give different answers.

“So it's just autocomplete?” Sort of, but don't let the word fool you. To guess well across every kind of text, a model has to pick up grammar, facts, tone and how an argument hangs together.

It doesn't only think one word ahead either. Anthropic looked inside Claude while it wrote a rhyming couplet and found it had picked the rhyme before starting the line, then wrote towards it. It writes one token at a time, but it can plan further ahead.

Your turn. Ask Claude to finish a sentence with just one word, then run it a few times. Does the word change?