How to Detect an AI Agent Cost Increase Caused by a Prompt or Model Change

What a cost regression is, and why nothing flags it

In this guide, a cost regression is a sustained increase in per-task token consumption or spend that follows a change to an agent's prompt, model, routing, retrieval, or tool configuration, while the agent keeps returning acceptable answers. Tests pass, error rates stay flat, and no alert fires. The first visible symptom is usually an aggregate invoice, which cannot say which agent, model, or tool call is responsible.

The cause is that these inputs change independently. One team expands a system prompt. Another swaps a model or a provider updates the version behind an alias. A retrieval change returns more context per query. A tool starts returning errors that trigger retries. Each change is reasonable in isolation. OWASP guidance names prompts, tools, memory, retrieval, policies, and model providers as separate change categories that warrant re-testing, which is a useful way to treat them: as independent change surfaces.

Four failure modes account for most cases. Prompt growth raises input tokens on every request. A model or version change can alter output length, tool-call behavior, or tokenization. A routing change sends traffic to a different model than before. A tool-call or retry loop multiplies the number of requests per task. OWASP frames unbounded loops mainly as a security risk, called denial-of-wallet, but the same mechanism applies when the trigger is an ordinary configuration change.

Building a baseline-and-compare approach tied to versions

Group requests by agent, prompt version, and model identifier. For each group, keep distributions rather than averages: input tokens per request, output tokens per request, requests per task, tool calls per task, and retry rate. Averages hide the long tail, and loops and prompt bloat often show up first at the high percentiles.

Use two comparisons. The first runs at release time. When a prompt, tool, retrieval, or model configuration changes, replay a representative task set and compare the new version's distributions to the previous version's before promotion. This fits the practice of re-testing after material changes and can act as a release gate. The second runs continuously in production. Compare a recent window against a trailing baseline for the same version, and alert on sudden changes in tokens, requests, or spend. Thresholds are team-specific, so set them from your own variance and review them as traffic mix changes.

Then read the shift against the change record. If it begins when the prompt version changes, suspect the prompt. If prompt and configuration versions are constant but the resolved model identifier changes, suspect a model or routing change. If both are constant and requests or tool calls per task rise, look at retrieval, tool behavior, or retries. This sorting only works if version tags are recorded on every request.

Two limits apply. A provider-side change under a stable alias is only distinguishable if the provider reports which model served the request, and some changes may leave no identifier difference. Baselines also assume comparable traffic, so a shift in task mix can look like a regression. Calls per task additionally require a task or session identifier that the application passes in, because individual requests do not carry one on their own.

• Release time: replay a fixed task set and compare version distributions before promotion
• Production: compare a recent window against a trailing baseline per version
• Attribution: match the onset of the shift to prompt version, resolved model identifier, or neither

Limiting the impact while you investigate

Detection takes time, and a loop can spend a lot before anyone reads a dashboard. Runtime limits bound the damage. OWASP guidance recommends retry, recursion, and chain-depth limits for tool-using inference flows, token and cost limits, rate limiting, and circuit breakers or kill switches for abnormal cost, latency, or tool-call spikes.

Granularity matters. Hard spending limits and billing alerts in a provider console are useful backstops, but they are aggregate controls. They do not tell you which agent, prompt version, or tool call is responsible, and they stop everything once reached. Limits applied per caller, per agent, and per tool call contain a single misbehaving configuration without halting unrelated work. For high-impact or unusual tool calls, escalation to a human for approval adds a deliberate pause instead of a silent retry.

There are tradeoffs. A limit set too tightly blocks legitimate long tasks, and one set too loosely only caps the loss. Limits also contain a regression without explaining it. They are a containment measure, and the baseline comparison is still what identifies the cause.

What an execution-path enforcement point can and cannot contribute

An enforcement point in the execution path sees each model request and tool call as it happens. That position supports two things. It can apply policy to the individual interaction, such as allow, redact, block, rate limit, or escalate for human approval, and it can produce a durable record of the decision. Trussed operates at this point. It enforces policy in the execution path at the level of individual interactions and tool calls, and the enforcement generates audit-ready evidence. For a cost regression, that means limits are enforced at runtime, not reviewed after the fact, and each enforced decision leaves per-request evidence you can use when attributing a change.

The boundaries are specific. Trussed does not provide end-to-end tracing of agentic workflows, and it does not provide model lineage. This page makes no claim to either. Reconstructing a task across agents, or establishing which model build served a request beyond what the request records show, is outside the scope described here. Version tags and task identifiers that the baseline approach depends on come from your application and configuration practice. Cost is treated here as a signal of unintended change in AI behavior, and this page makes no savings claim.

When you evaluate any enforcement or evidence layer for this problem, ask whether it can record the fields above per request, whether limits apply per interaction and per tool call instead of only as provider-level budgets, which actions can be blocked or escalated, whether those decisions are retained as durable evidence, and what the layer explicitly does not cover.