Technical Guide

How to Write an AI Agent Vendor Runtime Evidence Request

An AI agent vendor runtime evidence request asks for specific artifacts, tool-call logs, permission enforcement records, agent identity attestations, and policy violation records, that document what an agent actually executed, rather than relying on static documentation such as SOC 2 reports or architecture diagrams that describe intended design.

Core Artifact Categories to Include

A complete runtime evidence request should specify the following artifact types, each covering what the agent did, not only what it was designed to do.

  • Tool-call logs showing request, permission check, and outcome for each agent action
  • Permission enforcement records distinguishing allowed from denied actions
  • Agent identity attestations showing which credential or identity context executed an action
  • Policy violation or denial records, not only successful tool-call logs
  • Disclosure of which MCP (or equivalent protocol) version and authorization model the vendor implements

Runtime Evidence Is Not the Same as Compliance Documentation

Security teams evaluating AI agent vendors often default to the same questionnaire used for any SaaS vendor: SOC 2 reports, architecture diagrams, and written control narratives. These documents describe intended behavior and, in the case of SOC 2 Type II, the operating effectiveness of controls sampled over a defined review period. They do not show continuous, action-level records of what an agent did at runtime. An AI agent can be designed with scoped permissions and still execute actions outside that design if enforcement fails at the tool-call layer. Runtime evidence closes this gap by showing actual execution records rather than design intent. NIST SP 800-53's Audit and Accountability control family already establishes expectations for audit record content, generation, review, and retention in traditional systems. A runtime evidence request applies the same logic to AI agents, asking vendors to produce the equivalent records for tool calls, permission checks, and agent identity rather than assuming existing SOC 2 or architecture artifacts already cover this layer.

Why Standard Security Questionnaires Miss Agent-Specific Risk

Generic vendor security questionnaires ask whether access controls exist, not whether they were enforced on a specific action. This distinction matters for AI agents because OWASP's Top 10 for LLM Applications identifies excessive agency and insecure tool or plugin design as major risk categories, recommending scoped permissions and activity logging as mitigations rather than static policy statements. Meaningful tool-call governance evidence generally requires three linked elements per action: the request the agent attempted, the permission or policy check performed against that request, and the resulting outcome, whether allowed, denied, or modified. A questionnaire response confirming that permission controls exist says nothing about whether this three-part record is actually captured for a given tool call. MITRE ATLAS and OWASP's GenAI Security Project guidance on agentic threats both point to tool misuse and agent identity confusion as documented attack patterns, which gives buyers a concrete basis for scoping evidence requests around specific, known failure modes rather than general assurances.

Structuring the Evidence Request

Drafting Guidance for Security Engineers

  • Align log field and retention requirements with your organization's existing NIST SP 800-53 AU-family requirements so vendor evidence fits existing audit processes.
  • Distinguish between evidence a vendor can produce today and roadmap commitments, since granular per-tool-call logging capability still varies across agentic AI platforms.
  • Where MCP is in use, confirm the specific protocol version and whether authorization is enforced within the protocol itself, since earlier MCP implementations left authentication and authorization to the calling application rather than the protocol.
  • Treat NIST's AI RMF as a governance reference, not a logging schema. It recommends documentation and governance practices but does not prescribe specific runtime log fields, so buyers must specify their own field-level requirements.

Practical note

Specify retention and field-level requirements explicitly in the request itself. Vendors often confirm that logging exists without confirming that it meets your organization's audit and retention standards.

Runtime Evidence Categories to Request

These four categories summarize the artifact types covered above and can be used directly as a request checklist with vendors.

Tool-call logs

Per-action records of requests, parameters, and outcomes.

Permission enforcement records

Evidence that policy checks occurred before execution.

Agent identity attestations

Which credential or identity context executed each action.

Policy violation records

Denied or blocked tool-call attempts, not only successes.

Frequently Asked Questions

Is a SOC 2 report sufficient evidence for an AI agent vendor?

No. SOC 2 reports, particularly Type II, assess control operating effectiveness using sampled evidence over a review period. They are not continuous runtime logs and do not demonstrate tool-call-level enforcement on their own.

What if a vendor cannot produce tool-call-level logs?

Many current agent platforms log conversation-level prompts and responses but not granular per-tool-call records distinguishing authorized from denied actions. Treat this as a documented gap and a roadmap item to track, not an assumption of equivalent coverage.

Does using MCP guarantee runtime evidence availability?

No. MCP standardizes how agents connect to tools, and its specification has added an OAuth 2.1-based authorization framework. Evidence availability still depends on the vendor's specific implementation and logging choices.

How long should runtime logs be retained?

Retention should align with your organization's existing audit and accountability requirements, such as those described in NIST SP 800-53's AU control family, and should be confirmed explicitly rather than assumed from general log existence.

Validate Vendor Claims with Runtime Evidence

Trussed AI provides runtime governance for enterprise AI agents, including policy enforcement, audit logging, and agent identity and permission controls that generate the artifacts security teams need to evaluate vendor claims.

See Runtime Governance in Action