Skip to content
Tokens & temperature · Lesson 1 of 8

What is a token?

bits~/tokens-and-temperaturelesson 1/8 · 0% →
◆ tutor
A model doesn’t read letters or words. Before it sees your prompt, a tokenizer chops the text into pieces called tokens, and each piece becomes a number. Type anything below and watch where the cuts land. Spaces usually stick to the start of the next word.

tokenize "Unbelievably, the cat fetched."

[Un][believ][ably][,][ the][ cat][ fetched][.]
1844 31141 2915 11 279 8415 42118 13
8 tokens · 30 characters
◆ tutor
Unbelievably became three tokens, but the is just one. Common words are cheap; rare ones get split. Which of these is most likely a single token?