Skip to content
anythingai.space
How LLMs Workintermediate12 min · 8 steps

Transformer Architecture

One decoder-only transformer, end to end: embeddings, the pre-norm block, RoPE, GQA, SwiGLU and the LM head — and what changed between GPT-2 and 2026.

[b, seq, d]Training: logits at every position →cross-entropy lossGeneration: last position only →sample → append → repeatToken IDs[batch, seq]Embeddinglookup tableBlocks × Nattn + FFNLM head→ vocab sizeLogits
datatoken
1 / 8
Key ideaStep 1 of 8

Text in, next-token scores out

A decoder-only transformer — the shape behind GPT-3, Llama and most open models — is a short pipeline. Token IDs look up vectors in an embedding table, flow through N identical blocks, get a final RMSNorm, and the LM head projects each vector to one score per vocabulary entry: the logits.

In training, logits at every position feed a cross-entropy loss. In generation only the last position matters: sample one token, append it, repeat.

InsightGPT, Claude and Gemini configurations aren't public; the decoder-only shape is widely assumed for closed frontier models.

Lesson 4 of 6 in How LLMs Work