How to Reduce LLM Input Token Costs by Controlling System Prompts and Chat History

Where input tokens come from

A request to a model is assembled from several components, and each scales differently. The system prompt and other in-context instructions are prepended to the rest of the input. Because the model keeps no state between calls, these instructions travel with every request. Their size is constant per call, so a long system prompt adds the same overhead to every turn.

Chat history behaves differently. In most multi-turn applications the full transcript is resent with each new message. Input size per call grows with every turn, and total input across a conversation grows faster than the number of turns. Long sessions and long-running agents are where this compounds.

Retrieved context and tool output are the least predictable components. Retrieval returns whatever the query matches, up to whatever limit is configured. Tool calls return payloads such as query results, documents or API responses, and these are frequently appended to the context unchanged. Both channels also carry third-party content, which brings access-control and indirect prompt injection concerns in addition to size.

Before changing anything, instrument requests so input tokens are attributed to each of these four components per call, per application and per model. Without that breakdown, teams tend to trim the most visible component, usually the system prompt, while history or tool output is the real driver.

Mitigation techniques and their tradeoffs

Each technique constrains a different component and fails in a different way. The table summarizes what each addresses and what to watch for.

• Prompt consolidation: Targets the system prompt. Merge duplicated rules, remove obsolete examples and shorten wording. Risk: a rewrite can silently change or drop a required instruction, so re-evaluate behavior after every edit.
• History windowing: Targets history. Send only the most recent turns. Simple and predictable. Risk: the model loses earlier facts, user constraints and, if instructions live in the transcript, policy context.
• Summarization: Targets history. Replace older turns with a condensed summary. Retains more meaning than windowing. Risk: summaries lose detail, can introduce errors, and can carry sensitive values forward.
• Selective retrieval: Targets retrieved context. Return fewer, better-ranked passages and filter by relevance and permissions. Risk: too aggressive a cutoff removes the passage the answer needed.
• Provider prompt caching: Targets repeated prefixes. Providers that offer caching reuse a stable leading portion of the prompt. It depends on keeping static content first and variable content last. Behavior and eligibility rules differ by provider and model.
• Tool output limits: Targets tool results. Cap size, extract needed fields and drop raw payloads after use. Risk: truncating a result mid-structure can produce misleading input.

Apply the techniques in sequence

Order matters because earlier steps are lower risk and make later steps easier to evaluate. Start with changes that do not remove information the model sees, and move to lossy changes only where measurement shows they are needed.

• 1. Measure per component: Record input tokens for system prompt, history, retrieved context and tool output on each call. Establish a baseline for each application and model.
• 2. Consolidate the system prompt: Remove duplication and stale content. Separate required policy instructions from stylistic guidance so they can be managed differently.
• 3. Structure for prompt caching: Place static instructions at the start and per-request content at the end. Check the provider's current documentation for eligibility rules, since they vary and change.
• 4. Window history: Set a turn or token limit on transcript sent per call. Always exclude policy instructions from the window.
• 5. Summarize what the window drops: Summarize older turns only where continuity matters. Treat the summary as new input subject to the same redaction rules.
• 6. Narrow retrieval and tool output: Tune retrieval limits and cap tool results. Apply input filtering to third-party content, not only to user messages.

Keeping required instructions and policy intact

The main hazard in context reduction is not lost conversation detail. It is losing instructions the application must always follow, such as data-handling rules, disclosure requirements or restrictions on what the model may do. Naive truncation cuts from the oldest content, and instructions placed early or embedded in the transcript are exactly what gets cut first.

The design response is to partition the context. Keep required policy instructions in a protected region that windowing, summarization and consolidation never modify, and reduce only the conversational and retrieved regions. Do not let a summarizer rewrite policy text, and do not let a consolidation pass alter it without review.

Protecting that region has a second reason. System prompts are a known target of prompt extraction attacks, which attempt to reveal hidden instructions in a model's context. Reduction pipelines, summaries and debug logs should not expose or copy protected instruction content into places users or other systems can read.

Context also affects answer quality, so any reduction needs evaluation. Test representative conversations before and after each change, document the results, and keep the history needed for evaluation under an explicit retention policy. That retained data needs its own access control and review.