Inference & Servingadvanced14 min · 9 steps
Serving Infrastructure
From one GPU box to an inference fleet: gateways, KV-aware routing, model parallelism, autoscaling, streaming failover and an engine chosen per workload.
requesterrorsignaltokenheavy work
ProblemStep 1 of 9
One GPU box is not a service
A single engine replica answers requests — until traffic grows, a tenant floods it, or the model simply doesn't fit. Llama 3.1 405B needs about 810 GB for BF16 weights alone, before any KV cache.
The job has shifted from “serving a model” to orchestrating an inference fleet: routing, parallelism, isolation, scaling and failover.
Lesson 6 of 9 in Inference & Serving