Case Studiesadvanced14 min · 8 steps
Eval-Gated CI/CD for an AI Product
Make every PR earn its merge button: golden sets, calibrated LLM judges with confidence intervals, and per-axis gates under a $40-per-PR budget.
requesterrorheavy workresponsedata
metrics are illustrativeProblemStep 1 of 8
A “small” prompt change cost a $4M renewal
A RAG-plus-agent answer bot shipped a prompt tweak that quietly regressed answers about one contract type. The customer noticed before the team did, and a $4M renewal walked.
The post-incident review found three gaps: the change never ran against the contract-specific test set, the spot-check LLM judge had drifted by 11 points unnoticed, and a 2-day rollback took 9 days because there was no safe-to-revert baseline.
Watch outConstraints for the fix: under $40 of model spend per PR, about $15K a month in total, and p95 PR-to-merge under 90 minutes including eval.
← Previous lessonMulti-Tenant Fine-Tuning PlatformNext lesson →Customer-Specific Distillation Pipeline
Lesson 18 of 20 in Case Studies