Inference & Servingintermediate13 min · 9 steps
On-Device and Edge Deployment
Run models on laptops, phones and edge boxes: know the runtime layers, why Ollama isn't a server, how quantization makes models fit, and when local beats an API.
requesterrordata
metrics are illustrativeProblemStep 1 of 9
Not every model has to run in someone else's cloud
Local and edge inference is a real target in 2026. Lean local for regulated data that can't leave the box, offline or air-gapped operation, a latency floor (no 50–200 ms network round trip), and steady high volume where owned hardware beats per-token pricing.
Stay on APIs for frontier quality, spiky demand, low volume or thin ops capacity. The usual answer is a hybrid within one product.
Watch out“Privacy” alone no longer decides it: APIs offer zero-data-retention tiers — but some frontier tiers now require retention or safety monitoring. Read retention terms per model.
Lesson 9 of 9 in Inference & Serving