Words become tokens

Last lesson said the model guesses one token at a time. So what is a token?

Before Claude sees your prompt, a tokenizer chops the text into pieces from a fixed menu. The menu is big (modern models carry on the order of a hundred thousand pieces) but fixed, chosen so that any text can be written from it. Each piece has an id number, and the numbers are all the model ever sees. It doesn't see the letters inside them.

The cuts and ids above are illustrative. Every model family has its own tokenizer, Claude's included, so real splits differ a little. Type your own text to see roughly where cuts land.

Three things to notice. Common words get one token each. Rarer words get split into familiar chunks. And the space usually sticks to the front of the next word.

Why should you care? Two reasons. Limits and prices are counted in tokens, not words. And tokens explain some odd mistakes: ask a model to count letters and it has to work from chunks like “straw” and “berry”, not from letters.

That's enough tokens for now. The Tokens & temperature course goes deeper: counting them, weird splits, and how the odds turn into a choice.