Words become tokens

Last lesson said the model guesses one token at a time. So what is a token?

A tokenizer maps text to a sequence of integer ids from a fixed vocabulary, on the order of a hundred thousand entries, learned so that frequent strings get one id and anything else can be spelled from smaller pieces, down to bytes. The model is trained and run on those ids only. It never receives characters, which is why spelling, counting letters and exact string manipulation are off-distribution for it.

The split and the ids above are illustrative; Claude's tokenizer is not public, and it changed with Claude 4.7: Anthropic says the newer tokenizer yields roughly 30% more tokens for the same text than before. Never estimate from another vendor's tokenizer. Count with POST /v1/messages/count_tokens before a request, and read usage.input_tokens and usage.output_tokens on every response; that is what you are billed for and what fills the context window.

Patterns worth knowing. Common words are one token. Rare words, names and identifiers split into several. The leading space belongs to the token (“ straw”), so “straw” mid-sentence and “Straw” at a line start are different ids. Code, JSON and non-English text tokenize worse than English prose, so the same information costs more tokens.

Two consequences for building. Budgets, rate limits and prices are in tokens, so measure, don't guess. And tasks that depend on characters (count the r's, reverse this string, exact byte offsets) are the wrong shape for the model: do them in code and hand the model the result.

That's enough tokens for now. The Tokens & temperature course goes deeper: counting them, weird splits, and how the odds turn into a choice.