Skip to content
anythingai.space
Memory & Stateintermediate11 min · 8 steps

Short-Term Context Management

Keep the context window lean, cacheable and append-only: paged KV memory, prefix caching, sliding windows, summaries and selective dropping.

APPLICATION WINDOWsystem rulestool schemasturns 1–20tool resultsnew turnreply buffer64K app limitModel window: the hard limit (e.g. 1.05Mon GPT-6.1 Sol)Application window: your budget forlatency and cost (e.g. 64K)Buffer zone: room for the reply, thinkingtokens included
tokendatarequest
1 / 8
ProblemStep 1 of 8

Two windows, and the smaller one fills first

The model context window is the hard architectural limit (1.05M tokens on GPT-6.1 Sol, for example). The application context window is a budget you set, such as 64K, because attention cost grows with context and quality degrades well before the hard limit.

Every turn adds history and tool output. Soon the next message no longer fits, and you still need a buffer zone for the reply. On reasoning models that buffer must include thinking tokens, which count against max_tokens.

Watch outPlan the budget before you hit it: decide what gets dropped, summarised or offloaded, and in what order.

Lesson 2 of 6 in Memory & State