How to Run an AI Vendor Security Review in 5 Business Days

Why Standard Vendor Risk Questionnaires Fall Short

Most enterprise vendor risk programs rely on questionnaires built for traditional SaaS: data handling, encryption at rest and in transit, incident response history, and access controls at the account level. These questions remain relevant for AI vendors but leave a gap. Providers of AI agents, LLM APIs, or agentic infrastructure introduce a different risk surface: an agent that can authenticate independently, call external tools, and take actions with consequences beyond simple data access.

The OWASP Top 10 for Large Language Model Applications identifies risks such as excessive agency and insecure plugin or tool design that a generic questionnaire will not surface. A vendor can pass a standard security review and still grant its agents broader permissions than the task requires, or lack any mechanism to constrain tool-call behavior at runtime. Reviewing an AI vendor therefore requires supplementing existing vendor risk processes with AI-specific technical questions, not replacing them. NIST's AI Risk Management Framework, and its Generative AI Profile, offer a useful structure for this: organizing review activity around governing, mapping, measuring, and managing AI-specific risk rather than treating it as a separate track from standard vendor due diligence.

Five Evaluation Areas Generic Questionnaires Miss

Focus the AI-specific portion of the review on the control areas that traditional questionnaires rarely cover in enough depth.

A Day-by-Day Structure for the Review Window

Use the five-day window as an operational cadence. Front-load evidence requests so architecture, controls, and hands-on validation have enough time later in the week.

1. Day 1: Documentation intake

   Issue a focused request list covering attestations, architecture, agent identity, MCP or integration configuration, sample audit logs, and retention/export policy. Confirm scope of any SOC 2 report relative to AI or agent features.

2. Day 2: Architecture and identity review

   Trace how agent identity is provisioned and managed, where tool-call decisions are made, and how integration layers connect to external systems. Flag missing separation between model reasoning and tool-execution authority.

3. Day 3: Tool-call governance and runtime controls

   Evaluate least-privilege scoping at the tool-call and data-access level, allow-listing practices, and whether runtime policy can intercept and validate tool calls after deployment.

4. Day 4: Hands-on permission validation

   Validate stated boundaries with practical checks where access allows. Confirm that agents cannot exceed documented scopes and that audit logs capture decisions and tool outputs in a usable form.

5. Day 5: Decision and residual risk

   Consolidate findings into a documented go, no-go, or conditional-approval decision. Record rationale, accepted residual risk, owners, and follow-ups before the window closes.

Documentation to Request on Day One

Requesting the right documentation early determines whether the remaining four days are usable. A SOC 2 report based on the AICPA Trust Services Criteria remains a standard form of third-party attestation, but security teams should confirm explicitly whether the report's scope covers the AI or agent components of the service, or only the surrounding infrastructure. A report that predates the vendor's agentic features provides limited assurance about agent-specific controls.

Architecture diagrams should show how agent identity is provisioned, where tool-call decisions are made, and how MCP or equivalent integration layers connect to external systems. Vendors using MCP should be able to describe server-side configuration in specific terms: authentication method, transport security, and whether tool access is allow-listed or open by default. Finally, request a sample of audit logs covering agent actions and tool invocations, along with stated retention period and export capability. A vendor unable to produce any of these on request is a signal in itself, independent of what the documents ultimately show.

Technical Controls to Evaluate

Use the following checklist when reviewing architecture materials, configuration evidence, and hands-on validation results.

• Agent identity issued and managed separately from human user accounts
• Least-privilege permission scoping applied at the tool-call and data-access level, not just account level
• MCP server security, where applicable: authentication method, transport encryption, and tool allow-listing
• Runtime policy enforcement capable of intercepting and validating tool calls after deployment
• Structured, exportable audit logs covering agent decisions and tool outputs, distinct from standard application logs
• Architectural separation between model reasoning and tool-execution authority

Governance Decisions Within the Review

Keep the compressed review defensible under later audit by deciding process rules up front, not at the moment of approval.

• Map findings to NIST AI RMF functions (Govern, Map, Measure, Manage) to keep the review structure defensible under later audit
• Treat AI-specific findings as an extension of existing vendor risk management, not a separate parallel process
• Build in a conditional-approval path for low-risk outstanding items rather than forcing a binary decision under deadline pressure
• Document decision rationale and any accepted residual risk in writing at the time of approval
• Define an escalation path in advance for high-severity findings, such as unscoped agent permissions or missing audit logs