How LLMs Workintermediate13 min · 9 steps
Attention Mechanisms
How tokens look at each other — queries, keys and values, the causal mask, and the head-sharing tricks (GQA, MQA, MLA) that decide your KV-cache bill.
datarequestresponse
ProblemStep 1 of 9
Meaning lives in other tokens
In “The animal didn't cross the street because it* was too tired”, the vector for it* means nothing on its own. To understand it, the model has to look back and decide that animal — not street — is the relevant word.
Attention is the mechanism that lets every position gather information from every other position directly, instead of passing messages down a chain.
InsightDistributed-systems analogy: an RNN is message passing along a chain; attention is pub/sub where every node can query every other node.
Lesson 3 of 6 in How LLMs Work