Tool-Call Denial Rate Monitoring for AI Agents in Banks

Why the metric needs a precise definition

In April 2026 the OCC, Federal Reserve Board and FDIC issued revised interagency model risk management guidance (OCC Bulletin 2026-13, Federal Reserve SR 26-2). As of November 2026, the OCC bulletin states that it rescinds prior model risk management guidance. SR 11-7 is therefore historical context, though its view of ongoing monitoring remains a useful reference: confirm that a system performs as intended, and verify that any extension beyond its original scope is valid. The revised guidance describes ongoing monitoring as evaluating whether a model performs as expected as products, exposures, activities, clients and data change, including performance outside established thresholds.

A denial rate supports that purpose only if it is defined consistently. A workable definition is: denied tool calls divided by total attempted tool calls, per agent and time window. Use attempted calls as the denominator rather than sessions or tasks, because one task can produce many calls and a session-based ratio hides concentrated denials. No guidance prescribes these choices, so record them as a design decision that validators can review.

Track outcome types separately. Allow, redact, block, rate-limit and escalate are different decisions. Counting rate-limited or escalated calls as denials, or ignoring them, changes the metric without any change in agent behavior. Report denials as one outcome among several, and report escalations with their eventual human decision.

Source the metric from the enforcement layer, not from refusals in the model's response. A refusal in generated text does not undo an action already taken, and it does not show what policy actually permitted.

Choose segmentations that make the rate interpretable

An aggregate rate is ambiguous. A change could reflect prompt manipulation, drift, a policy edit or a legitimate workflow change, and only segmentation separates them. Start with agent, tool and denial reason, since reason mix is an established early-warning signal. Add policy rule, data class and escalation outcome where your enforcement records support them.

• Agent: Shows which deployment is changing and lets you compare behavior against its approved scope.
• Tool or MCP server: Isolates changes caused by a new tool, a modified server or an expanded permission set.
• Policy rule and version: Attributes a shift to a specific rule or to a policy change rather than to agent behavior.
• Data classification: Shows whether denials concentrate where sensitive data is involved.
• Denial reason: A change in reason distribution can precede a working bypass even when the total rate is flat.
• Escalation outcome: Records whether human reviewers approved or rejected escalated calls, supporting override analysis.

Baseline first, then set thresholds

Establish a baseline of expected behavior per agent, tool and parameter pattern before setting any threshold. Tracking which tools an agent calls, how often, in what order and with which parameters gives you a reference against which deviations, unusual tool combinations and atypical sequences become visible.

No source supplies a universal numeric threshold, and you should be skeptical of any that claims to. Calibrate thresholds, reporting frequency and escalation to the use case: the consequences of an error, the agent's autonomy, expected variability and rate of change. A customer-facing agent with write access to account systems warrants tighter bands and faster review than an internal summarization agent with read-only tools. SR 11-7 similarly tied monitoring frequency to the nature of the model and the magnitude of risk.

Annotate the time series. Mark every policy version change, tool or scope change and approved workflow change on the metric history. Without annotations, a step change after a policy edit looks identical to drift. Define two review triggers: a deviation from baseline beyond the agreed band, and a sustained change that persists after annotated changes are accounted for.

Read both high and low rates

A rising denial rate can mean prompt injection attempts, an agent reaching beyond its approved scope, a policy that is stricter than intended, or a tool change that broke an expected call pattern. Sudden shifts in refusal rates can indicate manipulation, drift or environmental change, which is why the rate alone is not a conclusion.

An abnormally low or flat rate deserves equal attention. It can mean policy is too permissive, a rule stopped matching after a tool or parameter change, enforcement is not covering a new tool, or an attack is succeeding. Shifts in the approval rate or reason distribution often precede a working bypass, so a rate that quietly drops to zero warrants the same scrutiny as a spike.

A stable rate is not proof of control. Offsetting changes can cancel out, such as more denials on one tool and fewer on another. Review segment-level movement and reason mix alongside the headline figure, and pair the metric with other indicators such as override, exception and escalation rates.

Route anomalies and place the metric in model risk management

Define the routing before the first alert fires. A deviation tied to a recent, documented policy or scope change goes to the control owner for confirmation. An unexplained deviation, a concentrated rise in one agent or data class, or a drop toward zero on a high-consequence tool goes to second-line review. Anything suggesting manipulation or an unapproved scope extension escalates through the bank's incident process. Record each disposition, because documented analysis of overrides is part of ongoing monitoring.

In governance documentation, position denial rate inside the existing ongoing-monitoring program under the revised model risk management guidance, rather than creating a parallel framework. The NIST AI RMF offers a clean mapping: the MEASURE function covers quantitative monitoring, and its outcomes feed the MANAGE function for response. State plainly that no supervisory statement specifies this metric. It is a practitioner control-effectiveness indicator, and its limits (denominator choice, policy changes, outcome mix) belong in the methodology note.