bits~/tokens-and-temperaturelesson 1/8 · 0% →
◆ tutor
A model doesn’t read letters or words. Before it sees your prompt, a tokenizer chops the text into pieces called tokens, and each piece becomes a number.
Type anything below and watch where the cuts land. Spaces usually stick to the start of the next word.
tokenize "Unbelievably, the cat fetched."
[Un][believ][ably][,][ the][ cat][ fetched][.]
1844 31141 2915 11 279 8415 42118 13
8 tokens · 30 characters
◆ tutor
Unbelievably became three tokens, but
the is just one. Common words are cheap; rare ones get split.
Which of these is most likely a single token?