Sampling Runtime Evidence in an Annual AI Agent Review Under Revised Model Risk Guidance

Why annual agent reviews depend on sampling

Agent behavior cannot be established from pre-deployment testing alone. What an agent does depends on the prompts, tools and data it meets in production, so an annual review has to examine what actually happened. A production agent can generate more interaction and tool-call records than any team can read, which makes sampling unavoidable. The question is whether the sample supports the conclusions the review draws.

The guidance basis has changed. The Federal Reserve (SR 26-2) and the OCC (Bulletin 2026-13) issued revised model risk management guidance on April 17, 2026. The earlier SR 11-7 called for periodic review of each model at least annually, more often if warranted, to determine whether it is working as intended and whether validation activities are sufficient. Reference the revised guidance as the current basis, as of November 2026, and keep the earlier wording as context for existing review calendars.

The OCC's revised guidance also notes that practices appropriate for one banking organization may be inappropriate for another with a different risk profile or model purpose. That principle applies directly to sampling. The plan should be tied to the agent's use case and risk, and the rationale should be written down.

Define the population before choosing a method

Statistical inference requires a defined population. Specify the agents in scope, the review period and the record types: interaction records, tool-call records and enforcement decisions. Treat each record type as its own frame, because a single interaction can produce several tool calls and the unit of review differs between them.

Then test completeness. Reconcile enforcement record counts against an independent source, such as tool-call volumes from the systems the agent called, across the whole review period. A gap means the frame is incomplete, and any sample drawn from it describes only the records that survived. Record the extract date and the counts, so the population is fixed at a point in time.

Spread the review across the full period rather than a single window. Agent usage changes over time, and long-term tracking of the extent and nature of agent use may be necessary. A sample drawn only from the last quarter says little about the first three.

Stratify by policy outcome, tool and data sensitivity

Simple random sampling of a whole population is dominated by routine allowed calls. Stratification fixes that. The two most useful dimensions are policy outcome (allow, redact, block, escalate) and the sensitivity of the tool or data involved. Each stratum can then be sampled at a rate that fits its risk, and results can be reported per stratum.

Rare high-risk events need separate handling. Blocks and human escalations may be few enough that reviewing every item is practical. If so, review all of them. If volumes are too large, select a judgmental set weighted toward sensitive tools or unusual patterns. The OCC Handbook, in the context of loan examinations, draws the key distinction. Judgmental sampling is not statistically based, so its results can inform conclusions but cannot support an inference about the entire population. Statistical sampling does allow that inference. The Handbook does not prescribe one methodology, and the application to agent records is by analogy.

Where the plan mixes both approaches, report them separately. State plainly that a judgmentally selected set of escalations supports findings about those items only.

Allowed and redacted records deserve attention too. A review that samples only blocks cannot detect a policy that should have blocked something and did not. Sample allowed calls in sensitive strata to test for that.

Size samples and fix selection rules

There is no settled formula for agent populations, so sample sizes rest on assumptions your institution owns and documents: the confidence level, the tolerable deviation rate, the expected deviation rate and the stratum size. Attribute sampling fits well, because each record either conforms to policy and intent or does not.

A general illustration helps with rare strata. If a random sample of 300 records shows no deviations, the upper bound on the true deviation rate is roughly 1 percent at 95 percent confidence. A tolerable rate tighter than that needs a larger sample. State the tolerable rate before drawing the sample, and let it drive the size.

Selection must be reproducible. Record the population extract date, stratum definitions, the random seed or selection method, exclusions and any replacement rules. Evaluations of language model systems are sensitive to model versions, prompts, sampling settings, data selection and exclusion rules, and endpoints can change. Exact retrospective replay is therefore difficult. Contemporaneous runtime records are the reliable basis, and the selection settings must be kept alongside them.

Document the rationale and state what the evidence supports

The sampling memo is part of the review record. It should cover the population definition and reconciliation result, stratification logic, sampling type per stratum, size assumptions, selection parameters, exclusions, exceptions found and the conclusions each result supports. Internal audit findings related to models should be documented and reported to the board or its delegate, and the plan should be reviewable by objective, informed parties as part of effective challenge.

Be explicit about scope. Samples of interactions and tool calls support conclusions about those events. They do not establish end-to-end workflow traces, model lineage or transaction-level governance. Trussed enforces policy in the execution path at the interaction and tool-call level (allow, redact, block, rate limit, escalate for human approval) and generates audit-ready evidence from that enforcement, which gives reviewers a population to sample from. Trussed does not provide transaction-level governance, model lineage or end-to-end agentic workflow tracing, so those questions need other evidence. Insurance, a regulated financial sector, is where Trussed is deepest. For other institutions, the same sampling discipline applies and should be tested against their own evidence.