Memory & Stateintermediate11 min · 9 steps
Semantic Caching
Reuse whole answers for equivalent questions: the embed-search-threshold pipeline, false hits, scoping, and when a provider prompt cache already wins.
requesterrorheavy workdataresponsesignal
ProblemStep 1 of 9
Same question, a dozen phrasings
Support traffic asks the same thing over and over, but rarely in the same words. An exact cache (Redis, Memcached) keys on a hash of the query string, so one typo or rephrasing is a miss.
Every miss pays for a full LLM call: seconds of latency and the full token bill, for an answer you already generated yesterday.
Lesson 5 of 6 in Memory & State