Skip to content
anythingai.space
Case Studiesadvanced14 min · 8 steps

Eval-Gated CI/CD for an AI Product

Make every PR earn its merge button: golden sets, calibrated LLM judges with confidence intervals, and per-axis gates under a $40-per-PR budget.

prompt tweakregressionNever run on the contracttest setJudge drifted 11 points,nobody noticed2-day rollback took 9 daysREQUIREMENTS28 engineers~50 PRs/week≤$40/PRmerge p95 <90min≈$15K/monthauditor signoffDeveloper PR~50 AI PRs / weekShip to mainCustomerregulated industry−$4M
requesterrorheavy workresponsedata
metrics are illustrative
1 / 8
ProblemStep 1 of 8

A “small” prompt change cost a $4M renewal

A RAG-plus-agent answer bot shipped a prompt tweak that quietly regressed answers about one contract type. The customer noticed before the team did, and a $4M renewal walked.

The post-incident review found three gaps: the change never ran against the contract-specific test set, the spot-check LLM judge had drifted by 11 points unnoticed, and a 2-day rollback took 9 days because there was no safe-to-revert baseline.

Watch outConstraints for the fix: under $40 of model spend per PR, about $15K a month in total, and p95 PR-to-merge under 90 minutes including eval.

Lesson 18 of 20 in Case Studies