How to Run an AI Agent Control Design Effectiveness Review

Design Effectiveness vs. Operating Effectiveness for AI Agents

Control design effectiveness and control operating effectiveness are distinct evaluation steps under established audit frameworks. Design effectiveness asks whether a control, as specified, is capable of preventing or detecting a defined risk if it functions as intended. Operating effectiveness asks whether that control actually performed as designed over a period of time. COSO's Internal Control-Integrated Framework and AICPA's SSAE 18 standards both require design adequacy to be assessed before any sampling of operating performance begins.

This sequencing matters for AI agents because many organizations skip directly to observing agent behavior in production, effectively testing operating effectiveness without ever confirming that the underlying control design was sound. If the design itself is inadequate, no amount of operating effectiveness testing will reveal a durable control. A permission model that grants an agent broad tool access by default is not a control that failed to operate correctly; it is a control that was never designed to enforce least privilege in the first place. Distinguishing these two failure modes is the foundation of a defensible review and directly affects how findings are reported to audit committees or regulators.

Why AI Agents Require an Adapted Review Approach

Traditional IT control reviews assume relatively static access control lists and deterministic system behavior. AI agents break both assumptions. Agents exhibit non-deterministic behavior, meaning the same input can produce different reasoning paths and different tool invocations across executions. Dynamic tool-call execution means an agent's effective permission scope can change at runtime based on its own reasoning, not solely on a fixed entitlement assigned in advance.

This shifts the review's center of gravity away from static access control lists and toward the policy enforcement points that govern tool access as it happens. Architectures built on the Model Context Protocol (MCP) illustrate this shift directly. MCP separates the agent host, the protocol client, and tool-providing servers into distinct components, each of which must independently validate authorization. The MCP specification requires servers to validate access tokens and requires hosts to obtain user consent before granting agent access to tools and resources. A control design review must therefore identify where these enforcement checks physically occur and confirm that each one addresses a specific risk objective, rather than assuming a single access control layer covers the entire agent architecture.

Control Domains That Must Be Assessed

A complete review addresses five control domains:

• Identity architecture: distinguishes between the human principal initiating a request, the agent's own service identity, and any delegated or impersonated identity used when the agent calls a downstream tool.
• Permission scoping: evaluated at the granularity of individual tools and resources, since role-based entitlements designed for human users rarely map cleanly onto an agent's dynamic invocation patterns.
• Tool-call governance: requires identifying the policy enforcement point architecture, whether centralized gateway, MCP server-side validation, or client-side guardrails, before design adequacy can even be assessed.
• Session and token lifecycle management: a distinct control surface under MCP-based implementations, covering issuance, scoping, expiration, and audience restriction to prevent token passthrough or confused-deputy scenarios between clients and servers.
• Auditability: depends on logging architecture capturing sufficient context, including intent, the specific tool called, parameters passed, and the outcome, to support later operating effectiveness testing.

NIST SP 800-53's Access Control and Audit and Accountability families provide baseline objectives applicable to each of these domains, even though they were not written with autonomous agents in mind.

Review Methodology

Reviewers map each identified risk objective to the specific control designed to mitigate it, then verify that design through walkthroughs of documentation and live configuration before any operating effectiveness sampling begins. This sequencing keeps the review focused on structural adequacy rather than on incidental observations of agent behavior.

Evidence and Documentation Requirements

Reviews must produce artifacts, not informal assessments. Internal audit and risk committees typically require documented evidence that design was assessed as a distinct step from operating effectiveness in order to support attestation or SOC-type reporting. For AI agent reviews, this means retaining:

• The permission matrix used to scope tool access.
• The policy definitions enforced at each policy enforcement point.
• Configuration records for MCP authorization flows, including token audience restrictions.

ISO/IEC 42001 requires organizations to define and document roles and responsibilities for AI system risk controls, and review documentation should reference those assignments directly rather than treating governance as implicit. NIST's AI RMF Govern function similarly expects organizations to demonstrate accountability structures for AI risk decisions, which should be reflected in the review's sign-off and escalation process.

Practical Guidance for Reviewers

Treat identity, permissions, tool-call governance, session and token lifecycle, and auditability as five separate lines of inquiry rather than a single access control checklist. Confirm that documented design matches actual system configuration through direct walkthrough before relying on any operating effectiveness evidence, and retain the underlying artifacts so findings can be defended to audit committees, regulators, or external assessors.