Preventing an AI Agent from Autonomously Escalating to a More Expensive Model

How an agent ends up on a different model

An agent rarely needs to "decide" to escalate. Several ordinary mechanisms can change which model serves a request, and each one can sit outside the code the owning team reviewed.

The first is direct selection. If an agent holds credentials or a client that can reach several models, a model parameter in a request can name a larger one. The second is framework behavior: configuration, fallback chains or retry logic can switch models without a change in the application's own code. The third is tool calls. A tool or MCP server the agent invokes may call its own model, so the agent's assigned model is not the only one in play. The fourth is delegation. A sub-agent or downstream service may be configured with different model access than its parent.

Delegation matters because weak scoping and inherited roles are a documented route to privilege escalation in agent systems. NIST's agent identity tenets address this with attenuation: as authority passes to sub-agents or downstream services, scope narrows, so a downstream entity never holds more authority than its parent. Applied to models, a delegated call should never receive a broader model allowance than the agent that made it.

Prompt instructions do not close these paths. Agent architectures generally combine trusted instructions and task data in a single input, so an instruction such as "use only the small model" shares a channel with content the agent cannot fully trust. NIST guidance on agentic threats also advises against trusting prompt instructions from users or other agents. A prompt can express intent, but it cannot enforce it.

Where the control can sit

The same policy has different reliability depending on where it is applied. The question for each location is whether it sees every model call, including those made by tools and sub-agents, and whether it can stop a call before it executes.

• Agent code: Easy to write and close to the logic. It binds only the code it lives in, so tools, sub-agents and framework fallbacks can fall outside it, and the agent's own owners can change it.
• Framework configuration: Centralizes settings for one framework. It covers only calls made through that framework and depends on the framework behaving as configured.
• Application layer: Can apply business rules before a request leaves the application. Calls made by tools or delegated components may never pass through it.
• Allowlist at a routing point: Constrains the traffic that reaches it. A model-name allowlist alone usually lacks agent, task and approval context, and it does not see calls routed around it.
• Execution path: Evaluates each interaction and tool call as it happens, with identity and task context, regardless of which component made the call. This is the most reliable point of control, provided every model call actually traverses it.

Write a model-access policy per agent and per task

A policy that can be enforced per call needs attributes that are available at request time. Start with a default of no model access, then grant narrowly, in line with zero standing privilege and task-bounded least privilege.

Keep the policy readable by the team that owns the agent. Each rule should state who, which model, for what, under what limit and what happens otherwise.

• Agent identity: Give each agent an identity distinct from human users and shared service accounts, with credentials that can be revoked.
• Task context: Grant model access per task, not per agent for all time. A summarization task and a code-review task can have different allowances.
• Permitted models: List models by exact identifier. Some commenters on NIST's agent identity work also proposed model family as policy metadata, which can simplify rules for related models.
• Escalation conditions: State when a larger model is acceptable, such as a specific task type or request class, and when it requires approval. Which calls are high-impact is your team's decision.
• Rate limits: Cap calls per agent, per task and per model tier, so an unexpected pattern is contained even when each call is individually permitted.
• Delegation scope: Require that sub-agents and tool-initiated calls inherit no more than the calling agent's allowance.

Enforce four outcomes in the execution path

Each model call or tool call is evaluated against the policy and receives one of four outcomes.

Allow: the call matches a permitted model, task and rate, and proceeds. Block: the call names a model outside scope, and it never reaches the model. The agent receives a defined refusal it can handle, rather than a silent substitution to a different model. Rate limit: the call is in scope but exceeds a threshold, so it is delayed or refused. NIST guidance describes throttles and rate limits as blast-radius controls, which fits an agent that loops or fans out calls. Escalate for human approval: the call falls into a category the policy reserves for a person, and it waits until an approver decides.

Two design choices deserve care. Decide the failure mode up front: if the policy decision cannot be made, the default should be to deny. And keep approval requests specific, showing the agent, task, requested model and reason, so approvers decide on facts rather than approving in bulk. Per-action approval is recommended for high-impact actions, and overuse erodes it.

Trussed is one example of this approach. It enforces policy in the execution path across models, agents, tools and MCP servers, at the level of individual interactions and tool calls, with outcomes that include allow, block, rate limit and escalate for human approval. It operates with 100+ models under one policy plane.