Deployment & Deliveryintermediate11 min · 10 steps
Canary Releases
Send a small, consistent slice of real users to a new version, compare health and answer quality against the current one, and promote in steps — or abort in seconds.
users on new100%
escalation29%
re-ask14%
requestsignalerrordata
metrics are illustrativeProblemStep 1 of 10
The new version doesn't crash. It's just worse.
A new system prompt passed every offline gate. Shipped to everyone, it would raise no errors and no latency alarms — and yet answer worse, in ways the golden set didn't cover.
Many LLM regressions look like this: no exception, just more confused users, more re-asked questions, more escalations to a human.
Watch outDashboards built for web services stay green while an LLM gets quietly worse.
Lesson 14 of 20 in Deployment & Delivery