What is a token?

A model doesn’t read letters or words. Before it sees your prompt, a tokenizer chops the text into pieces called tokens, and each piece becomes a number.

Type anything below and watch where the cuts land. Spaces usually stick to the start of the next word.

Unbelievably became three tokens, but the is just one. Common words are cheap; rare ones get split.