AI Audit Evidence Without Vendor Data Access: What Runtime Enforcement Can Prove

Evidence is a product of enforcement, not a by-product of logging

Regulated enterprises are asked to show what an AI system was allowed to do and what it actually did. Many AI security tools answer by sending prompts, responses and tool-call content to a vendor-operated service. That creates third-party data exposure, residency conflicts and procurement friction before the first audit begins.

The alternative starts from where evidence is created. If a policy decision is made in the execution path, at the moment a model call or tool invocation occurs, the record of that decision is a direct product of enforcement. It does not have to be reconstructed later from application logs that were written for other purposes.

This matches the direction of NIST guidance on agent identity, which describes an append-only, tamper-evident body of evidence covering the agent's identity, the delegated intent and the resulting actions. It also calls for each tool invocation, API call or resource access to be verified at runtime against current state, context and risk. NIST further notes that language models process data and instruction as one token stream, so separation between them has to be enforced by architecture, with policy enforcement around the agent and between it and the systems, data and credentials it touches. Enforcement points are therefore the natural source of evidence.

Which evidence needs content inspection and which does not

The useful split is between evidence that describes a decision and evidence that depends on reading the content being decided on. The first kind can usually be recorded as metadata. The second requires inspection, and the question becomes where inspection runs. The table below is a working classification, not a statement of what any regulation requires.

• Policy decision records: Policy identifier, version, subject, target and decision. Typically metadata only.
• Allow, block, rate limit and escalation outcomes: Outcome, timestamp, and the human approver for escalations. Typically metadata only. Blocked and rejected actions are worth keeping as evidence in their own right.
• Tool-call records: Tool or API invoked, caller identity, time and result. Parameters may contain sensitive content, so decide per field whether to store them.
• Redaction outcomes: Proof that a redaction occurred requires inspecting content. Store the fact and category of redaction without the original text.
• Content-based detections: Output filtering and pattern detection depend on reading content. Confirm which environment performs the inspection.
• Code provenance context: Origin, licence and security context of code. Depends on analysing the code itself.

How evidence is generated, stored and retained in the customer cloud

In a self-managed deployment, the enforcement component runs in the customer's cloud account, in line with traffic between applications, models, agents, tools and MCP servers. Each interaction or tool call is evaluated against policy, the outcome is applied, and an evidence record is written as part of that same step. Content inspection needed for redaction or detection runs in the same environment, so nothing has to leave it for the decision to be made or recorded.

Storage is the second half of the design. The evidence store should sit in infrastructure the customer controls, in the region the customer chooses, with retention set by the customer. Retention periods differ by regime, so a fixed vendor default is a weak fit. NIST's GenAI Profile draft suggests a long-term retention policy that keeps full history for auditing and investigation, which implies that retention has to be configurable and that storage capacity has to be planned. Evidence volume can be substantial when every interaction is recorded.

There are trade-offs against a vendor-hosted model. The customer operates the deployment, sizes the storage, manages upgrades and owns availability of the evidence store. In exchange, the customer decides who can read records, where they reside and how long they live. NIST's cloud forensics work lists data integrity, metadata integrity and tenant confidentiality as challenges in shared environments, which are good reasons to ask where evidence lives and who can reach it.

Establishing integrity without relying on the vendor

An evidence record is only useful if an auditor can trust it. The sourced pattern is append-only storage with tamper evidence and an unbroken chain of custody. In a customer-hosted deployment, that means the customer's own controls enforce write-once behaviour, restrict deletion and record administrative access to the store.

Cryptographic techniques add independent verifiability. Chaining records by hash or signing them lets a reviewer detect gaps, reordering or alteration using only the customer's data and public verification steps. NIST's GenAI Profile draft discusses cryptographic verification for content provenance. Applying the same technique to enforcement decision records is a design choice to confirm with the vendor, not an assumed feature.

The test is simple. An auditor should be able to sample a decision, trace it to a policy version, confirm the record has not changed, and do this without asking the vendor to produce or vouch for the data. If any step requires vendor-side access, the evidence depends on the vendor in practice, whatever the architecture diagram says.

Treat framework references with care. Several NIST sources on this topic are profiles, drafts or blog guidance, so they describe good practice and not binding audit requirements. Confirm the record-keeping and retention obligations of each applicable regime, such as HIPAA, NAIC model guidance, NYDFS or the EU AI Act, with your compliance and legal teams, and keep retention configurable so it can follow them.

An example of in-the-path enforcement, and where its evidence stops

Trussed is one example of this pattern. It enforces policy in the execution path across models, agents, tools, MCP servers, AI sites and coding assistants, applying allow, redact, block, rate limit or escalation for human approval. Enforcement is at runtime, not reviewed after the fact, and it produces audit-ready evidence mapped to HIPAA, NAIC, NYDFS, the EU AI Act and NIST. It can be deployed self-managed in the customer cloud, with configurable retention and data residency, and it records code provenance context covering origin, licence and security.

It has run in continuous production in a regulated enterprise for more than 18 months, at 99.95% uptime on live production traffic. It has 100+ models under one policy plane and generates 30+ GB of audit evidence per day. That volume is a reminder to size storage and retention deliberately.

The limits matter as much as the capabilities. The scope is individual interactions and tool calls. The evidence does not describe model provenance, model lineage or dataset provenance, and it is not business transaction identity or end-to-end tracing of an agentic workflow. If an auditor asks about those, you need other sources of evidence. Stating that boundary up front is part of making the evidence credible.