Skip to content
anythingai.space
Deployment & Deliveryintermediate11 min · 9 steps

Load Balancing

Spread long, uneven LLM streams across healthy instances — and across model deployments — without hot spots, lost context or cut-off answers.

busiest pod18 streams
p959.5s
UsersLoad balancerround robinPod A18 streamsPod B4Pod C11
requestheavy worksignaltokendataerror
metrics are illustrative
1 / 9
ProblemStep 1 of 9

Equal requests, unequal load

The load balancer uses round robin: each pod gets the same number of requests. But LLM requests last anywhere from a second to several minutes — a quick lookup versus a long agent loop with streamed output.

A snapshot of 33 open streams (illustrative): pod A holds 18, pod B 4, pod C 11. Pod A is slow for everyone on it, and that pod drives the service's p95.

Lesson 13 of 20 in Deployment & Delivery