Why Vendor Claims Are Not Enough
Enterprises evaluating AI agent platforms typically rely on vendor documentation, sales demonstrations, and security questionnaires. These sources describe intended behavior but rarely reveal how a system actually responds when an agent attempts something it should not be allowed to do. Two vendors can both claim least-privilege enforcement and audit logging while differing substantially in how those controls behave under adversarial conditions.
A bake-off addresses this gap by treating the evaluation as a technical exercise rather than a checklist review. Security engineers define a fixed set of test scenarios, run them identically against each vendor's platform, and record observed behavior rather than stated capability. This produces comparable evidence rather than self-reported assurances, and it surfaces architectural differences, such as whether authorization is enforced at the protocol layer or only within individual application logic, that are difficult to detect through documentation alone.
What to Test: Identity, Permissions, and Tool Calls
Three categories of runtime behavior matter most in an agent security evaluation. The first is agent identity: whether the platform maintains an identity for the agent that is distinct from the end user it acts on behalf of, and whether actions taken by the agent can be traced back to that identity after the fact. This distinction matters because agents often act autonomously across multiple steps, and without a bound identity it becomes difficult to attribute a given tool call to a specific agent instance or session.
The second category is permission scoping. Some platforms enforce access through static role assignments set at configuration time. Others evaluate permissions dynamically, considering context such as the specific tool being called, the data involved, or the session state. Bake-off testing should include scenarios where an agent's assigned scope changes mid-session, to observe whether the platform re-evaluates permissions or continues operating on stale authorization.
The third category is tool-call interception: whether the platform can inspect and block a tool invocation before it executes, or whether it can only record the call after execution has already occurred. This distinction has direct security consequences, since after-the-fact logging does not prevent an unauthorized action; it only documents it.
Core Test Categories at a Glance
Identity Binding
How agent actions are tied to accountable identities distinct from end users.
Permission Scoping
Static roles versus dynamic, context-aware policy evaluation at runtime.
Tool-Call Interception
Whether requests are checked before execution or only logged after.
Audit Traceability
Granularity and completeness of logs for incident investigation.
Scoring Auditability and Incident Traceability
Runtime security controls are only useful if failures and near-misses can be investigated after the fact. Scoring criteria for a bake-off should evaluate whether audit logs capture individual tool calls or only session-level summaries, since per-call granularity is necessary for reconstructing what an agent actually did during an incident. Logs should also be assessed for immutability and export capability, since logs that can be altered after generation, or that cannot be extracted for independent review, are of limited value to an incident response process.
Aligning scoring categories to NIST AI RMF functions (Govern, Map, Measure, and Manage) gives the evaluation a structure that covers both technical enforcement and organizational accountability, rather than focusing solely on whether a control exists. This also helps translate bake-off results into language that governance and compliance stakeholders can use during vendor selection.
Recent Standards Context: MCP Authorization
The Model Context Protocol has become a common reference point for evaluating agent-to-tool authorization, since its specification defines security considerations including token handling, confused deputy risks, and explicit user consent for tool invocations. The authorization draft has been revised multiple times in the past year, including changes to dynamic client registration and resource indicator requirements intended to prevent token misuse across servers.
Because the specification is still evolving, evaluators should confirm which version of the MCP authorization model a vendor claims to support, and verify that claim against current documentation rather than relying on older vendor materials. Where a vendor does not implement MCP-compliant authorization, the bake-off should include direct questions about how equivalent protections, such as scoped tokens and consent checkpoints, are implemented instead.