How LLMs Workintermediate14 min · 9 steps
The Inference Pipeline
From prompt to streamed answer: why prefill and decode behave so differently, how sampling and speculation work, and which latency and memory numbers to watch.
tokenheavy workrequest
Key ideaStep 1 of 9
Generation is a loop, one token per lap
LLMs generate autoregressively: run the model over everything so far, get a probability for every vocabulary entry, pick one token, append it, and go again.
The loop stops on an EOS token, a max-tokens limit, or a custom stop sequence. Everything about serving cost and latency comes from how many laps you run and how expensive each lap is.
tokens = tokenize(prompt)
for _ in range(max_tokens):
logits = model.forward(tokens)
next_token = sample(logits[-1])
if next_token == EOS: break
tokens.append(next_token)Lesson 6 of 6 in How LLMs Work