Why Standard Vendor Questionnaires Fall Short for AI Suppliers
Traditional third-party risk questionnaires were built for software that behaves predictably: fixed inputs, fixed outputs, static permissions. AI systems, particularly those with agentic capabilities, do not fit that model. A model's output can vary between runs. An AI agent may call external tools, chain multiple actions together, or make decisions without a human in the loop at each step. None of this is captured by questions designed for traditional SaaS procurement, which focus on uptime, data encryption, and basic access logs.
This creates a gap. Procurement teams often complete a vendor security review, receive a SOC 2 report, and move forward, without ever asking how the AI system behaves once it is granted access to internal systems, what happens when it calls a tool it was not expected to use, or how its decisions are logged and attributable. The 40 questions below are organized to close that gap by treating AI vendor evaluation as a distinct discipline from general software vendor management.
Category 1: Data Handling and Storage (Questions 1–8)
Data handling questions establish where enterprise data goes once it enters the vendor's system, and whether it can leave that boundary in ways the enterprise does not control. Ask the vendor to specify whether customer data is used for model training or fine-tuning, whether data is retained after a session ends, and for how long. Ask which subprocessors or downstream model providers receive access to enterprise data, and whether that list is disclosed and kept current.
Also ask how data is segmented between customers in a multi-tenant environment, what encryption is applied at rest and in transit, and what the process is for data deletion on contract termination. For AI systems that process regulated or sensitive data categories, ask whether the vendor can identify and restrict that data from entering model context in the first place, rather than relying on after-the-fact review.
Category 2: Model Governance and Behavior (Questions 9–16)
Model governance questions address how the vendor controls what the model can produce and how changes to that model are managed over time. Ask how frequently the underlying model is updated, whether the vendor notifies customers before a model version change, and what testing occurs before an update is pushed to production. Ask whether the vendor can explain, at a reasonable level of detail, how the model was evaluated for the specific use case being procured.
Ask what guardrails exist to constrain model outputs, such as content filtering, output validation, or refusal behavior for out-of-scope requests, and whether those guardrails are configurable by the customer or fixed by the vendor. If the system generates decisions that affect customers, employees, or business outcomes, ask how those decisions can be reviewed, appealed, or reversed.
Category 3: Runtime and Agent Behavior (Questions 17–24)
This category is the most commonly underdeveloped section of AI vendor questionnaires, and it is where agentic systems introduce risk that traditional software never had to address. An AI agent that can call tools, query databases, or take actions on behalf of a user is functioning more like a semi-autonomous system than a static application, and it needs to be assessed accordingly.
Ask the vendor to list every tool, API, or system the agent can access, and whether that list is fixed or can expand without customer visibility. Ask whether tool calls require approval before execution, or whether the agent acts autonomously once granted access. Ask what happens when the agent encounters an ambiguous or unexpected situation: does it halt, escalate, or attempt an action anyway. Ask whether agent-to-agent communication is possible in the vendor's architecture, and if so, how those interactions are constrained and logged. These questions determine whether the enterprise is deploying a bounded tool or an open-ended actor.
Category 4: Access Control and Identity (Questions 25–30)
Access control questions establish who, or what, can act within the AI system and under what constraints. Ask how the vendor implements authentication for both human users and any automated agents or service accounts. Ask whether permissions follow a least-privilege model by default, or whether the system grants broad access that must be manually restricted.
Ask how the vendor handles credential and secret management for any systems the AI connects to, and whether those credentials are scoped narrowly to specific functions. For agentic systems specifically, ask whether each agent has a distinct, auditable identity, or whether multiple agents share a single set of credentials, which makes it difficult to attribute actions to a specific process later.
Category 5: Monitoring, Logging, and Auditability (Questions 31–36)
Monitoring questions determine whether the enterprise will have visibility into what the AI system actually did, after the fact and in real time. Ask what activity the vendor logs by default: prompts, outputs, tool calls, decisions, and errors. Ask how long logs are retained, whether they are exportable to the enterprise's own monitoring systems, and whether log integrity is protected against tampering.
Ask whether the vendor provides alerting for anomalous behavior, such as an agent attempting an action outside its normal pattern, and whether that alerting is configurable by the customer. Ask how the vendor supports incident investigation, including what forensic detail is available if the enterprise needs to reconstruct a sequence of AI-driven actions after an incident.
Category 6: Compliance and Governance (Questions 37–40)
The final category addresses how the vendor's practices align with the enterprise's own governance obligations. Ask what internal AI governance processes the vendor follows for its own development and deployment practices, and whether those processes are documented. Ask how the vendor supports the enterprise's need to demonstrate oversight of AI systems to internal risk committees, auditors, or regulators.
Ask whether the vendor can provide evidence of testing for known AI-specific risks relevant to the use case, and how that evidence is documented. Ask what contractual commitments the vendor is willing to make regarding data use, model behavior, and notification of material changes, since verbal assurances during a sales process are not a substitute for enforceable terms.
The 40 Questions at a Glance
| Category | Questions | Core focus |
|---|---|---|
| Data Handling and Storage | 1–8 | Training use, retention, subprocessors, encryption, deletion |
| Model Governance and Behavior | 9–16 | Update cadence, evaluation, guardrails, decision review |
| Runtime and Agent Behavior | 17–24 | Tool access, approval workflows, escalation, agent-to-agent activity |
| Access Control and Identity | 25–30 | Authentication, least privilege, credential scoping, agent identity |
| Monitoring, Logging, and Auditability | 31–36 | Default logging, retention, tamper protection, alerting, forensics |
| Compliance and Governance | 37–40 | Internal governance, oversight evidence, contractual commitments |
How to Use This Questionnaire Effectively
The questionnaire is most useful when it is applied consistently across every AI vendor under evaluation, and when answers are treated as claims to be verified rather than assurances to be accepted. Route the runtime and agent behavior questions to whichever team will be responsible for the system once it is in production, since procurement and security staff may not have visibility into how the tool will actually be used day to day.
Practical tip
Where a vendor's answer is vague or aspirational, ask for supporting documentation rather than accepting a verbal or generic response. Specific, auditable answers are a stronger signal than confident but unverifiable ones.
What Happens After the Questionnaire
A completed questionnaire tells the enterprise what the vendor claims about its system. It does not, by itself, verify that those claims hold true once the AI system is deployed and operating against live enterprise data and systems. Vendor answers about tool access, agent autonomy, and logging need to be validated against what actually happens at runtime, particularly for agentic systems where behavior can be less predictable than the vendor's documentation suggests.
This is the point where questionnaire-based due diligence and runtime governance need to connect. Runtime governance for AI agents applies controls such as agent identity, permission enforcement, tool approval workflows, and audit logging directly at the point of execution, so that the assumptions established during vendor evaluation are enforced continuously rather than assumed to hold indefinitely. Procurement teams that pair a strong questionnaire with runtime enforcement reduce the risk that a vendor's stated practices diverge from its actual production behavior.