Skip to content
anythingai.space
Reliability & Safetyintermediate13 min · 9 steps

Reliability Patterns

Retry the right errors, stop hammering a dead provider, isolate workloads, set layered timeouts and degrade gracefully when everything fails.

THROWS AN ERROR429 rate limit429 spend captimeoutoutageretired modelcontext overflowmalformed outputLOOKS LIKE SUCCESSsame-ID quality change200 + stop_reason: refusalSDK default-model changeLLM clientProvider A
requesterrorsignalresponseheavy work
1 / 9
ProblemStep 1 of 9

LLM APIs fail loudly, and sometimes silently

Rate limits, spend caps, timeouts, outages, retired model IDs, context overflow and malformed output all throw. Two failures don't: a quality change under the same model ID, and a safety refusal that arrives as HTTP 200 with empty content.

Set an availability and latency target per tier, then design as if the provider will go down, because it will.

TierAvailabilityp99 latency
Critical (payments)99.99%< 3 s
Standard (support)99.9%< 10 s
Best effort (background)99%< 30 s
What changed
  • +Failure modes, loud and silent

Lesson 3 of 4 in Reliability & Safety