Skip to content
anythingai.space
Training & Adaptationbeginner12 min · 9 steps

Pretraining Basics

How a model soaks up language and knowledge from trillions of tokens, and why data, compute, memory and stability are the hard parts, not the objective.

CONTEXT SO FARThecatsatonthe?MODEL'S GUESS FOR THE NEXT TOKEN (ILLUSTRATIVE)mat0.62 ← true tokenfloor0.21sofa0.09moon0.01Decoder-only LLMlearns by prediction
datasignal
1 / 9
Key ideaStep 1 of 9

One objective: predict the next token

Most modern LLMs are decoder-only and trained with causal language modeling: given every token so far, predict the next one. The loss is cross-entropy, the negative log-probability the model gave the true next token.

That objective is almost embarrassingly simple. At trillions of tokens it produces grammar, world knowledge and the beginnings of reasoning.

Many current models add an auxiliary multi-token prediction (MTP) head, popularised by DeepSeek-V3. It strengthens the training signal and leaves behind draft heads that serving engines reuse for speculative decoding.

# minimise cross-entropy over the corpus
loss = -Σ log P(tok_i | tok_1 .. tok_i-1)
What changed
  • +Context tokens
  • +Next-token distribution

Lesson 1 of 8 in Training & Adaptation