Skip to content
anythingai.space
Inference & Servingintermediate13 min · 9 steps

Cost Optimization Playbook

Cut inference cost by routing to the cheapest capable model, keeping prefixes cached, picking a service tier per workload and knowing when self-hosting pays.

blended4$/1M
on Tier 10%
every requestAppTier 2 · midSonnet 5.5 · $2 / $10100%
heavy workrequest
metrics are illustrative
1 / 9
ProblemStep 1 of 9

Everything goes to the mid-tier model

Measure success as tokens per dollar at a given quality bar and latency SLO. The default setup fails that test: every request — greetings, extraction, hard reasoning — goes to the same mid-tier model.

At $2 in / $10 out and a 3:1 input-to-output blend, that's about $4 per million tokens for work a model 20× cheaper could do.

ComponentCost driverOptimization
ComputeGPU time ($/hr)Better utilization (batching)
VRAMKV cache sizeGQA, quantization
NetworkPayload sizeCompression, local serving
APIPer-token priceCaching, model choice, service tier

Lesson 7 of 9 in Inference & Serving