Skip to content
anythingai.space
Inference & Servingintermediate13 min · 9 steps

Batching Strategies

From static batches that wait for the slowest request to continuous batching, chunked prefill and fully disaggregated prefill/decode pools.

GPU busy45%
throughput1×
worst ITL30ms
FOUR BATCH SLOTS, ONE STATIC BATCHSlot 1A · 500 tokensSlot 2Bidle — waiting for ASlot 3CidleSlot 4DidleQueueE, F, G wait for the whole batchdecode steps →
requesterrorheavy worktoken
metrics are illustrative
1 / 9
ProblemStep 1 of 9

Static batching waits for the longest answer

Static batching (classic ML) starts a batch together and ends it together. LLM answers vary wildly in length, so if one user wants 500 tokens and another 5, the short request's slot sits idle for 495 steps.

Meanwhile new requests queue outside, waiting for the whole batch to finish.

Watch outThe “longest tail” problem: the batch runs at the speed of its slowest member, and short requests pay the highest latency.

Lesson 4 of 9 in Inference & Serving