How to Evaluate a Small AI Vendor's SOC 2, Pen Test, and Secure SDLC Evidence

Start by mapping artifacts to AI components

Small AI vendors often hand over three documents: a SOC 2 report, a pen test summary, and a set of secure development statements. Each one asserts something, but none asserts the same thing. The first step is to list what the vendor actually ships: model-serving infrastructure, prompt and data handling, tool and agent integrations, and third-party model dependencies. Then record, for each artifact, whether each component is inside its stated scope, outside it, or not mentioned.

The OWASP Artificial Intelligence Security Verification Standard (AISVS) is a useful reference for this step. It is a catalogue of testable requirements for AI-enabled systems, and OWASP lists evaluating AI vendors and third-party models in procurement as one of its intended uses. Its categories include infrastructure, configuration and deployment security; supply chain security for models; memory, embeddings and vector database security; orchestration and agentic security; and Model Context Protocol (MCP) security. If a vendor's evidence never touches a category that matches its product, that is a finding.

Request the underlying documents, not summaries. A desktop review of public information is a reasonable minimum, but it is not an assessment. Move to supplier-specific evidence requests wherever the desktop review leaves a component unexplained.

Reading a SOC 2 report for an AI vendor

A SOC 2 Type I report describes whether controls were suitably designed at a point in time. A Type II report also tests whether controls operated effectively over a stated period. Young vendors frequently have only a Type I. That is not disqualifying, but it is design evidence, not operating evidence, and it should be weighted accordingly.

The most common failure is a scope mismatch. The system description defines what the auditor examined. If it describes a corporate application platform and says nothing about model serving, prompt and data flows, or agent integrations, the opinion does not speak to the AI service you plan to use. Read the description before reading the opinion.

For a Type II, check the period and how long ago it ended. Controls can change after the audit window closes, so ask what the vendor has done since. Then read the exceptions closely, including management's responses, because the pattern of exceptions often says more than the clean parts of the report. Finally, look at carve-outs. If the cloud host or a model provider is a carved-out subservice organization, the auditor did not test those controls, and the report relies on assumptions about them that you need to verify separately.

• Report type and period: Type I or Type II, the audit window, and the time since it ended.
• System description: Whether it names the production service and its AI components.
• Trust services criteria: Security is the baseline; confirm whether confidentiality, privacy, or availability were chosen where your data and use case need them.
• Auditor opinion: Whether it is unqualified, and whether any qualification touches in-scope systems.
• Exceptions: What failed, how often, and whether the response was credible.
• Carve-outs and user controls: Which subservice providers were excluded and which controls the report expects you to operate.

Judging whether a pen test covers the AI attack surface

NIST SP 800-115 describes penetration testing as assessors mimicking real-world attacks to find ways around the security features of an application, system, or network. It also notes that no single technique gives a complete picture. A pen test is one input, not a verdict.

For AI systems, the OWASP AI Testing Guide adds that models learn, adapt, and fail in non-deterministic ways, creating risks that conventional security testing does not address. A generic web or network test may therefore be accurate and still silent on prompt handling, tool calling, or retrieval paths.

Several questions separate credible reports from thin ones. Who performed the test, and are they independent of the team that built the product? What methodology did they follow? The OWASP GenAI Red Teaming Manual, for example, describes eight phases from planning through reporting and remediation. Did the scope match what the vendor ships? OWASP distinguishes simpler systems such as chatbots and RAG applications from advanced ones with tool-calling agents, MCP architectures, or multi-agent workflows, and the scope should reflect which applies. Did it include APIs, model endpoints, prompt and tool-calling paths, and tenant isolation?

Then look at follow-through. Findings by severity matter less than what happened next. Ask for retest evidence showing that high-severity items were fixed and verified. If the vendor used an external AI red team provider, you can assess that provider against OWASP's published vendor evaluation criteria.

Testing secure SDLC claims against artifacts

Statements such as "we review all code" or "we manage dependencies" are easy to make and hard to evaluate. The NIST Secure Software Development Framework (SP 800-218) gives you a way to ask for proof. Practice PO.4.1 calls for defined criteria for software security checks, with examples that include recording approvals, rejections, and exception requests. The question to put to a vendor is simple: can you show the criteria, and can you show the records?

Reasonable artifacts from a small vendor include code review records showing who approved changes, a dependency inventory or SBOM, secrets-scanning configuration and results, a vulnerability tracker with ages and dispositions, and release approval records. For AI products, add how the vendor assesses and monitors third-party models and the broader model supply chain, which AISVS treats as its own category.

Draft SSDF language also points to retention policies for evidence and artifacts and at-least-annual review of security requirements. These are draft provisions, but they are sensible questions: how long does the vendor keep SDLC evidence, and when did it last review its requirements?

Keep the limits in view. The SSDF and its generative AI community profile address cybersecurity risk management only. Strong SDLC evidence says little about other AI risks, so it should not carry more weight than it can support.

Common gaps and how to respond

Some gaps cannot be closed by asking more questions. When they remain, the response should match the specific weakness rather than defaulting to rejection or blanket approval.

• Type I only, or a stale Type II period: Limit initial data classes, shorten the term, and require delivery of the next report by a stated date.
• Model provider or host carved out of SOC 2: Ask for that provider's own assurance, require notice of model or subprocessor changes, and flow security terms down contractually.
• Pen test omits agent, tool, or MCP paths: Require a scoped retest before tool access is enabled, and restrict or disable those integrations until then.
• No retest evidence for serious findings: Set remediation timelines for critical and high findings and require retest results on request.
• SDLC claims without records: Negotiate audit rights and vulnerability notification terms, and reduce the data and permissions the vendor receives.

What vendor evidence cannot give you

Even a strong package describes the vendor's controls at a point in time. It does not show what happens inside your own deployment on a given day: which data reaches the model, which tool calls an agent attempts, and what was permitted or stopped. For a regulated enterprise, that is a separate requirement, and it sits on the buyer's side.

This is where compensating controls become concrete. Trussed enforces policy in the execution path across models, agents, tools, and MCP servers, applying allow, redact, block, rate limit, or escalate-for-approval decisions to individual interactions and tool calls. The enforcement produces audit-ready evidence of what the AI was allowed to do and what it did. It does not replace vendor due diligence, and it does not assess the vendor's internal controls. It limits exposure when that assessment leaves residual gaps, and it gives auditors evidence from your own environment.

In practice, record each unresolved gap in the risk assessment together with its mitigation: a contract term, a restricted data class, a disabled integration, or a runtime policy. Revisit those entries when the vendor delivers a new report or retest.