Scoping starts with data flows, not the redaction tool
Auditors begin by scoping every path personal data can travel, rather than by examining the redaction tool alone. The goal is to inventory applications, agents, traffic paths, and data stores that can carry PII.
In LLM traffic, the paths in scope include:
- Prompts
- Context supplied to the model
- Tool calls
- Responses
- Logs, including traces
Testing design effectiveness
Design testing checks that the control is designed to cover the paths identified during scoping. Auditors confirm four things:
- Enforcement point: where redaction is applied to the traffic.
- Coverage: whether every in-scope path is addressed.
- Failure behavior: what the control does when it cannot perform as designed.
- Change control: how changes to the control are managed.
Testing operating effectiveness
Operating effectiveness testing shows whether the control works in practice across the audit period. Auditors rely on three techniques:
- Seeded synthetic PII: test data placed into traffic to confirm it is redacted.
- Bypass attempts: deliberate efforts to get PII past the control.
- Sampling: review of samples of traffic, logs, and traces.
Evidence auditors typically examine by traffic path
Auditors expect evidence for each path in scope, whether prompts, context, tool calls, responses, or logs. Typical evidence includes:
- Policy configurations
- Runtime enforcement logs
- Test results
- Proof that stored logs are themselves free of unredacted PII
Logs are a data store, and detection has limits
Stored logs are in scope as a data store in their own right. Auditors look for proof that logs are themselves free of unredacted PII, and they sample logs and traces as part of operating effectiveness testing. Log inspections are therefore part of the evidence set.
Assessing readiness and preparing evidence
To assess readiness, walk through the audit lifecycle above and confirm that each stage can be supported. Prepare configurations, enforcement logs, test results, and log inspections before the audit starts.