Where it learned: pretraining

A model is its parameters: billions of weights, initialised at random, so the untrained network's next-token distribution is roughly uniform.

Pretraining is self-supervised next-token prediction over a large text corpus. For each position, the model predicts a distribution over the vocabulary, the loss is the cross-entropy against the token that actually came next, and gradient descent nudges every weight to lower it. No labels, no questions, no answers: the text supervises itself.

Scale is the point. The loss starts near ln(vocab size), about 11.5 nats for a hundred-thousand-token vocabulary, and falls as the model absorbs token frequencies, then collocations, then syntax, then long-range structure. Scrub the run below; the numbers are illustrative, the curve is the real shape.

Nothing in that curve was programmed. Grammar, facts and style are whatever lowers next-token loss on the corpus, which is why a model is fluent in things nobody specified, and also why it absorbs the corpus's errors and biases just as readily.

The corpus, for Claude Haiku 5.5, per Anthropic's Transparency Hub: a proprietary mix of public internet data, public and private datasets, user data from feedback or explicitly permitted for training, and synthetic data generated by other models, with cleaning and de-duplication, and a training data cutoff of June 2026.

The model is orders of magnitude smaller than its corpus, so pretraining is lossy compression. High-frequency structure is stored precisely; a fact that appeared a handful of times is stored as a blurry tendency, and the model will produce a fluent token sequence over that blur. That is the mechanism behind hallucination (lesson 8), and the reason retrieval and tools exist: they put exact facts back into the context instead of relying on the weights.

What comes out of pretraining is a base model: a brilliant continuer of documents, but not yet something that answers you. Next lesson, it becomes one.