Stopping Context Window Growth from Raising Per-Turn Cost in Long AI Agent Sessions

Why per-turn token consumption grows in agent loops

A model call is stateless. In a multi-turn agent loop, the runtime rebuilds the full input on every turn: system prompt, tool definitions, the conversation so far, each tool call and its result, and any retrieved documents. The agent's work on turn 40 therefore pays for everything that happened on turns 1 to 39. Input tokens, latency and per-turn cost rise with session length even when each new step is small.

Tool output is usually the largest contributor. A single query result, file read or web fetch can exceed all the reasoning text around it, and once it is in the history it is resent on every later turn whether or not it is still relevant. Retrieved content behaves the same way. Oversized retrieved chunks also carry a quality risk: a long retrieved document can push the system prompt out of the model's effective attention window, which is why limits on the number and total size of chunks are recommended.

Treat this as an architecture and lifecycle problem. No single technique bounds context without a trade-off, so the practical question is which combination you can apply, in what order, and with what record of what each step removed.

Context management techniques compared

The techniques below are ordered roughly by implementation effort and risk. Each changes what the model sees, so each needs an evaluation against your own task set before rollout.

• Truncation or sliding window: Lowest effort. Drops the oldest turns. Failure mode: early constraints, instructions or decisions disappear without notice. Pin the system prompt and standing requirements outside the window.
• Tool-output limits and clearing: Cap result size at entry and replace stale results with a short placeholder. Targets the largest source of growth. Failure mode: the agent needs a cleared result later and must call the tool again, or proceeds without it.
• Summarization and compaction: Replaces older history with a model-written summary. Keeps continuity at lower token cost. Failure mode: lossy summaries omit details, flatten uncertainty or introduce errors that persist for the rest of the session.
• External memory and retrieval: Stores facts and artifacts outside the prompt and retrieves on demand. Bounds context well. Adds retrieval quality risk, a write path to validate, and a scoping requirement between users and sessions.
• Prompt caching: Reuses a stable, repeated prefix so repeated input is processed more cheaply. It reduces the cost of resending, not the size of the context. Edits earlier in the prompt generally reduce reuse, so summaries that rewrite the prefix can work against it. Confirm your provider's documented limits.
• Sub-agent context isolation: Delegates a bounded task to a separate context and returns only a compact result. Keeps the parent small. Failure mode: the returned result drops detail the parent needs, and multi-agent designs raise cost exposure if loops are unbounded.

Failure modes and how to evaluate them

Context reduction fails quietly. The agent keeps running, but it may have lost a constraint stated in turn three, a tool result that justified a later action, or an instruction that was only ever given once. Design for this explicitly.

Separate content that must never be reduced from content that can be. Standing instructions, policy constraints and user-stated requirements belong in a pinned section that truncation and summarization do not touch. Everything else is a candidate for clearing or compaction.

Evaluate with your own tasks. Run the same long-session scenarios with and without each technique, and check whether early constraints are still honored, whether the agent repeats tool calls it already made, and whether final outputs match. Include cases where a fact from early in the session is needed late. Compare latency and token counts alongside accuracy so a saving is not bought with silent degradation.

Treat tool output and retrieved content as untrusted until validated, including content that has been summarized. A summary of an injected instruction is still an injected instruction, now with less visible provenance. Scan retrieved chunks for injection patterns before they enter context, and validate and sanitize data before it is written to agent memory.

What a regulated enterprise must preserve

Every reduction technique answers the question "what does the model see?" differently from the full history. A regulated enterprise also has to answer the later question: what was the agent given, and what did it do? If a tool result was cleared, a history was summarized or a field was redacted, the record should show that, not just the final prompt.

Logging all agent decisions, tool calls and outcomes is the baseline. It does not by itself capture reductions, so that needs separate design. For each turn, retain what was removed or replaced, the rule or trigger that caused it, the summary text that replaced the original, and a reference to the original content under your retention policy where policy allows. Where content was redacted because it was sensitive, keep the fact and basis of the redaction rather than the sensitive value.

Stored memory should be trustworthy as a record, so use integrity checks on long-term memory and isolate memory and context between users and sessions. Shared or insufficiently scoped context can expose sensitive information from one task, user or agent to another, and isolation should be enforced by design rather than by prompt wording.

Runtime policy enforcement helps at the entry point. Trussed enforces policy in the execution path on individual interactions and tool calls, applying allow, redact or block decisions at runtime, not reviewed after the fact. That can keep sensitive or oversized tool output out of the context before it accumulates, and the enforcement itself generates audit-ready evidence of what the agent was permitted to do and what it did. It does not replace the agent runtime's own record of how it summarized or compacted history, and it is not end-to-end tracing of an agent workflow.