Before training, a model is a huge set of numbers called parameters, all set at random. Ask it anything and it guesses tokens at random too. Pure noise.
Pretraining fixes that with one exercise, repeated an enormous number of times. Take some real text. Hide the next token. Let the model guess. Then check.
The check gives a score called the loss: how surprised the model was by the real next token. High loss means it barely saw it coming. Then every parameter gets nudged a tiny bit, so the right token comes out a little more likely next time.
That's it. Guess, check, nudge, over and over, across a mountain of text. Drag through the training run below and watch one prompt go from noise to sense.
model output at step 0
Once upon aqz# lorp vv]
loss11.0
Lower loss means the model’s guesses for the next token got better. Drag to watch it learn.
The numbers are illustrative, but the arc is real. Roughly, it learns which tokens are common, then which ones go together, then grammar, then whole stories. Nobody wrote those rules in. They show up because they help the guesses.
What text? For Claude Haiku 5.5, Anthropic says it's a proprietary mix of public information from the internet, public and private datasets, some user data (from feedback, or that people allowed for training) and text written by other models, cleaned and de-duplicated first.
Here's a useful way to think about the result. The text is far bigger than the model, so it can't keep a copy of it all. It mostly keeps patterns, a bit like a lossy zip file of everything it read. The gist survives. Rare details get fuzzy. Hold on to that, because it explains a lot in lesson 8.
What comes out of pretraining is a base model: a brilliant continuer of documents, but not yet something that answers you. Next lesson, it becomes one.