Runtime Evidence Is Not the Same as Compliance Documentation
Security teams evaluating AI agent vendors often default to the same questionnaire used for any SaaS vendor: SOC 2 reports, architecture diagrams, and written control narratives. These documents describe intended behavior and, in the case of SOC 2 Type II, the operating effectiveness of controls sampled over a defined review period. They do not show continuous, action-level records of what an agent did at runtime. An AI agent can be designed with scoped permissions and still execute actions outside that design if enforcement fails at the tool-call layer. Runtime evidence closes this gap by showing actual execution records rather than design intent. NIST SP 800-53's Audit and Accountability control family already establishes expectations for audit record content, generation, review, and retention in traditional systems. A runtime evidence request applies the same logic to AI agents, asking vendors to produce the equivalent records for tool calls, permission checks, and agent identity rather than assuming existing SOC 2 or architecture artifacts already cover this layer.
Why Standard Security Questionnaires Miss Agent-Specific Risk
Generic vendor security questionnaires ask whether access controls exist, not whether they were enforced on a specific action. This distinction matters for AI agents because OWASP's Top 10 for LLM Applications identifies excessive agency and insecure tool or plugin design as major risk categories, recommending scoped permissions and activity logging as mitigations rather than static policy statements. Meaningful tool-call governance evidence generally requires three linked elements per action: the request the agent attempted, the permission or policy check performed against that request, and the resulting outcome, whether allowed, denied, or modified. A questionnaire response confirming that permission controls exist says nothing about whether this three-part record is actually captured for a given tool call. MITRE ATLAS and OWASP's GenAI Security Project guidance on agentic threats both point to tool misuse and agent identity confusion as documented attack patterns, which gives buyers a concrete basis for scoping evidence requests around specific, known failure modes rather than general assurances.
Structuring the Evidence Request
Drafting Guidance for Security Engineers
- Align log field and retention requirements with your organization's existing NIST SP 800-53 AU-family requirements so vendor evidence fits existing audit processes.
- Distinguish between evidence a vendor can produce today and roadmap commitments, since granular per-tool-call logging capability still varies across agentic AI platforms.
- Where MCP is in use, confirm the specific protocol version and whether authorization is enforced within the protocol itself, since earlier MCP implementations left authentication and authorization to the calling application rather than the protocol.
- Treat NIST's AI RMF as a governance reference, not a logging schema. It recommends documentation and governance practices but does not prescribe specific runtime log fields, so buyers must specify their own field-level requirements.
Practical note
Specify retention and field-level requirements explicitly in the request itself. Vendors often confirm that logging exists without confirming that it meets your organization's audit and retention standards.