Deployment & Deliveryintermediate11 min · 9 steps
Load Balancing
Spread long, uneven LLM streams across healthy instances — and across model deployments — without hot spots, lost context or cut-off answers.
busiest pod18 streams
p959.5s
requestheavy worksignaltokendataerror
metrics are illustrativeProblemStep 1 of 9
Equal requests, unequal load
The load balancer uses round robin: each pod gets the same number of requests. But LLM requests last anywhere from a second to several minutes — a quick lookup versus a long agent loop with streamed output.
A snapshot of 33 open streams (illustrative): pod A holds 18, pod B 4, pod C 11. Pod A is slow for everyone on it, and that pod drives the service's p95.
Lesson 13 of 20 in Deployment & Delivery