Compliance Guide

How auditors test PII redaction controls in LLM traffic

Auditors test PII redaction in LLM traffic by scoping every path personal data can travel, checking that the control is designed to cover those paths, and then testing operating effectiveness with seeded synthetic PII, bypass attempts, and sampling of traffic, logs, and traces. They expect evidence in the form of policy configurations, runtime enforcement logs, test results, and proof that stored logs are themselves free of unredacted PII. None of this guarantees an audit outcome, but it makes the control testable.

The audit lifecycle for PII redaction

An audit of PII redaction moves through four stages. The sections below follow the same order.

Audit lifecycle stages and what each one covers
StageWhat happens
1. ScopeInventory applications, agents, traffic paths, and data stores that can carry PII.
2. Design testConfirm the enforcement point, coverage, failure behavior, and change control.
3. Operating testRun synthetic PII, bypass attempts, and sampling across the audit period.
4. EvidenceCollect configurations, enforcement logs, test results, and log inspections.

Scoping starts with data flows, not the redaction tool

Auditors begin by scoping every path personal data can travel, rather than by examining the redaction tool alone. The goal is to inventory applications, agents, traffic paths, and data stores that can carry PII.

In LLM traffic, the paths in scope include:

  • Prompts
  • Context supplied to the model
  • Tool calls
  • Responses
  • Logs, including traces

Testing design effectiveness

Design testing checks that the control is designed to cover the paths identified during scoping. Auditors confirm four things:

  • Enforcement point: where redaction is applied to the traffic.
  • Coverage: whether every in-scope path is addressed.
  • Failure behavior: what the control does when it cannot perform as designed.
  • Change control: how changes to the control are managed.

Testing operating effectiveness

Operating effectiveness testing shows whether the control works in practice across the audit period. Auditors rely on three techniques:

  • Seeded synthetic PII: test data placed into traffic to confirm it is redacted.
  • Bypass attempts: deliberate efforts to get PII past the control.
  • Sampling: review of samples of traffic, logs, and traces.

Evidence auditors typically examine by traffic path

Auditors expect evidence for each path in scope, whether prompts, context, tool calls, responses, or logs. Typical evidence includes:

  • Policy configurations
  • Runtime enforcement logs
  • Test results
  • Proof that stored logs are themselves free of unredacted PII

Logs are a data store, and detection has limits

Stored logs are in scope as a data store in their own right. Auditors look for proof that logs are themselves free of unredacted PII, and they sample logs and traces as part of operating effectiveness testing. Log inspections are therefore part of the evidence set.

Assessing readiness and preparing evidence

To assess readiness, walk through the audit lifecycle above and confirm that each stage can be supported. Prepare configurations, enforcement logs, test results, and log inspections before the audit starts.

Prepare runtime evidence before the audit starts

Discuss how runtime policy enforcement and audit logging can support testable controls for LLM and agent traffic.

Talk to an Expert about audit evidence