Logging LLM Refusals as Audit Evidence

Two kinds of refusal that look alike

A refusal can originate in two places. The model can generate text such as "I cannot help with that", which is behavior of the model. Or an enforcement layer in the execution path can block, redact or escalate a request or a tool call according to a defined policy. To a user these may look the same. To an auditor they are different things.

A model-generated refusal shows that the model declined. It does not show that any policy was applied, because no control decision was enforced. A policy-enforced refusal is a control decision: a named policy evaluated a request from an identifiable actor and produced a decision that the system carried out. Only the second type can answer the auditor's questions about what was allowed or denied and whether it was enforced at runtime.

Do not separate the two by matching refusal phrases in the response. OWASP guidance advises against classifying answers by phrase matching, and models phrase refusals in many ways. Instead, record the decision source as an explicit field at the time of the decision, rather than inferring it from text afterward.

Define the refusal event record

A structured record is what turns a log line into evidence. NIST defines an audit log as a record providing documentary evidence of specific events, so each event should be self-explanatory when read alone. The fields below are a practical baseline. Confirm them against the framework you are being audited to, since each may expect additional items.

Use enumerated values for decision type and reason rather than free text. OWASP observes that sudden changes in approval rates or in the distribution of refusal reasons often precede a working bypass, which is only measurable if reasons are structured and queryable. Where you record test or review outcomes, use explicit categories such as violation observed, no violation observed, inconclusive and not applicable.

• Event identity and time: Unique event ID, timestamp from a trusted clock, and the system and environment that produced it.
• Actor and request context: The user, service or agent that made the request, the application, and the target model or tool.
• Decision source: Enforcement layer or model output. Never leave this to be inferred.
• Policy and decision: Policy identifier and version, the decision (allow, deny, redact, escalate), and a structured reason code.
• Scope of the action: Whether the decision applied to the prompt, the response, or a specific tool call, and whether any action had already occurred.
• Content reference: A pointer, digest or classification of the content, not the raw content itself unless it is required.

Sensitive content, integrity, retention and access

Prompts and responses often contain personal or confidential data. OWASP recommends logging security-relevant metadata and decisions while excluding credentials, secrets and unnecessary sensitive prompt or response content. The EU AI Act also states that data minimisation and data protection by design apply when personal data are processed. That sits in tension with keeping full content as evidence. A workable resolution is to keep the decision record rich and the content record minimal: store the decision, policy, actor and a digest or classification of the content, and place any raw content that must be retained in a separate store with tighter access and its own retention period.

For integrity, the record must be hard to alter unnoticed. Common techniques include append-only or write-once storage, hash chaining or signing of records, and separating the permissions of those who write logs from those who administer the systems being logged. Choose the technique your auditors and threat model require, and test that tampering is detectable.

Retention should be set per record class and reflect both uses OWASP names for audit trails: compliance and forensics. Decision records may need to outlive raw content. For EU high-risk AI systems, the regulation says systems must technically allow automatic recording of events over their lifetime, as of October 2026. Whether a given deployment is high-risk is a classification question to settle separately. Finally, restrict read access to the evidence store, log access to it, and keep it in a region that meets your data residency obligations.

How runtime enforcement supports this

Trussed enforces policy in the execution path across models, agents, tools and MCP servers, with decisions to allow, redact, block, rate limit or escalate for human approval. Because decisions are enforced at runtime, not reviewed after the fact, the event is produced by the component that made the decision. Trussed generates audit-ready evidence from that enforcement, mapped to frameworks including HIPAA, NAIC, NYDFS, the EU AI Act and NIST. It deploys self-managed in the customer cloud, with configurable retention and data residency.

The design principles above apply whichever tooling you use: capture at the point of decision, record the source and the policy, minimise content, and protect the record.