Inference & Servingintermediate13 min · 9 steps
Batching Strategies
From static batches that wait for the slowest request to continuous batching, chunked prefill and fully disaggregated prefill/decode pools.
GPU busy45%
throughput1×
worst ITL30ms
requesterrorheavy worktoken
metrics are illustrativeProblemStep 1 of 9
Static batching waits for the longest answer
Static batching (classic ML) starts a batch together and ends it together. LLM answers vary wildly in length, so if one user wants 500 tokens and another 5, the short request's slot sits idle for 495 steps.
Meanwhile new requests queue outside, waiting for the whole batch to finish.
Watch outThe “longest tail” problem: the batch runs at the speed of its slowest member, and short requests pay the highest latency.
Lesson 4 of 9 in Inference & Serving