Pretraining Basics
How a model soaks up language and knowledge from trillions of tokens, and why data, compute, memory and stability are the hard parts, not the objective.
One objective: predict the next token
Most modern LLMs are decoder-only and trained with causal language modeling: given every token so far, predict the next one. The loss is cross-entropy, the negative log-probability the model gave the true next token.
That objective is almost embarrassingly simple. At trillions of tokens it produces grammar, world knowledge and the beginnings of reasoning.
Many current models add an auxiliary multi-token prediction (MTP) head, popularised by DeepSeek-V3. It strengthens the training signal and leaves behind draft heads that serving engines reuse for speculative decoding.
# minimise cross-entropy over the corpus
loss = -Σ log P(tok_i | tok_1 .. tok_i-1)- +Context tokens
- +Next-token distribution
Lesson 1 of 8 in Training & Adaptation