Skip to content
anythingai.space
Inference & Servingintermediate11 min · 9 steps

PagedAttention

Borrow virtual memory from operating systems: store the KV cache in small fixed-size blocks, cut memory waste from 60–80% to under 4%, and share prefixes for free.

KV wasted70%
requests / GPU4
GPU MEMORY · CONTIGUOUS KV, RESERVED AT MAX LENGTHweightsA usedA reserved, emptyB usedB reservedfreeReserve 8,192 token slots, generate 10 → 99.9% of theblock wasted
requestdata
metrics are illustrative
1 / 9
ProblemStep 1 of 9

Reserve for the longest possible answer

Standard frameworks allocate one big contiguous buffer per request. Since you don't know how long the answer will be, you reserve for max_sequence_length — say 8,192 tokens.

If the user's answer is 10 tokens, 99.9% of that reservation is dead weight. That's internal fragmentation.

Lesson 5 of 9 in Inference & Serving