Inference & Servingintermediate12 min · 9 steps
Speculative Decoding
Guess several tokens cheaply, verify them all in one pass of the big model, and get 2–3× faster decode with exactly the same output.
per token50ms
tokens / pass1
tokenheavy workerror
metrics are illustrativeProblemStep 1 of 9
140 GB of reads for one 2-byte token
Decode is memory-bound. A 70B model streams ~140 GB of weights out of HBM to produce one token per sequence — and then does it all again for the next one.
The arithmetic units sit mostly idle. A pass that checked five tokens would cost about the same as a pass that produced one.
Lesson 3 of 9 in Inference & Serving