Inference & Servingintermediate13 min · 9 steps
Cost Optimization Playbook
Cut inference cost by routing to the cheapest capable model, keeping prefixes cached, picking a service tier per workload and knowing when self-hosting pays.
blended4$/1M
on Tier 10%
heavy workrequest
metrics are illustrativeProblemStep 1 of 9
Everything goes to the mid-tier model
Measure success as tokens per dollar at a given quality bar and latency SLO. The default setup fails that test: every request — greetings, extraction, hard reasoning — goes to the same mid-tier model.
At $2 in / $10 out and a 3:1 input-to-output blend, that's about $4 per million tokens for work a model 20× cheaper could do.
| Component | Cost driver | Optimization |
|---|---|---|
| Compute | GPU time ($/hr) | Better utilization (batching) |
| VRAM | KV cache size | GQA, quantization |
| Network | Payload size | Compression, local serving |
| API | Per-token price | Caching, model choice, service tier |
Lesson 7 of 9 in Inference & Serving