Request-level vs session-level AI evidence: what auditors actually need

Define the terms before testing anything

Neither "request-level" nor "session-level" evidence has a single settled definition, so an audit should fix the terms first. In this guide, a request is one interaction submitted to an AI system, with the response and any action that follows. A tool call is a single invocation of a tool, data source or external service by the AI. A session is a bounded series of requests, sometimes called a conversation, that may span many turns. NIST's evaluation materials use a similar idea of a session as one interaction between a tester and an application that can include multiple conversational turns.

The distinction matters because NIST SP 800-53 Rev. 5 treats two capabilities separately. Event logging records audit content such as an event description, a timestamp, source and destination addresses, user or process identifiers, and a success or fail indication. Session audit (AU-14) records the content of a user session and is implemented in addition to event logging. That layering is a useful model: event-level records carry the control evidence, and session records add context.

What each granularity can and cannot demonstrate

Start from the assertion you need to test, then ask which record could support it. The mapping below reflects practitioner judgment about control testing, not a prescribed standard. Session-level records are not weak evidence. They answer different questions.

• What the AI was permitted to do: Request-level: shows the permissions evaluated for that interaction or tool call. Session-level: shows what happened in the session, but may not show what was permitted at each step.
• Which policy was applied: Request-level: can carry the policy identifier and version for that call. Session-level: usually describes the session as a whole, so the policy in force at a given step may need to be inferred.
• What decision was enforced (allow, redact, block, escalate): Request-level: records the decision and any redaction or block action per call. Session-level: may show the final transcript without showing what was removed or stopped.
• What happened as a result: Request-level: records the outcome of the specific interaction. Session-level: shows the sequence and context around it.
• Who was involved and in what order: Request-level: identity and timestamp per call. Session-level: sequence, continuity and conversational context.

What the frameworks say, and what they leave open

The guidance available supports general points rather than a per-request mandate. NIST describes a body of evidence as objective, demonstrable and repeatable information recorded during the production and use of a system. It also notes that information given to auditors must be at a level and depth fit for the purpose of the review. The EU AI Act text refers to keeping records, making technical documentation available so compliance can be assessed, and, among logging capabilities, recording the start and end date and time of each use of the system. Which provisions apply to which systems is a question for counsel.

None of this says that every AI interaction must have its own record. The case for request-level evidence comes from the control being tested. If a policy is applied to each interaction or tool call, the auditor needs evidence at that level to confirm it was applied. Expectations under HIPAA, NAIC and NYDFS should be taken from the regulators' own materials.

This shapes sampling. If the control operates per interaction, define the population as requests or tool calls and sample from it, rather than sampling sessions and hoping the sampled sessions contain the relevant events. Sampling by session is reasonable for testing context, such as whether a conversation was handled consistently, but it is a poor fit for confirming that a specific policy decision was enforced. The request record should carry a session or conversation identifier so you can move from a sampled request to its surrounding context.

Retention, integrity, residency and export

Evidence that cannot be produced is not evidence. The NIST Generative AI Profile draft recommends a long-term retention policy that keeps the full history for auditing and investigation. It does not set a period, so retention should be set against the regulations that apply to the system. Confirm that the period covers the audit window and that records are not overwritten when storage or configuration changes.

Integrity is a separate question. An auditor should ask how records are protected from alteration, who can modify or delete them, and whether the record was written at the time of the event or assembled later. Evidence generated at the point where the policy is applied is generally easier to rely on than a record reconstructed afterward from aggregate logs.

Privacy cuts the other way. NIST notes that session auditing can reveal information about individuals and advises developing it with legal counsel. NIST also describes holding commercially sensitive or privacy-impinging evidence in a separate evidence repository. One practical design keeps searchable metadata for sampling in one place and sensitive prompt or response content in a controlled store. Residency should be confirmed for both. Finally, test export: can the organization produce a defined sample, with its linked context, in a usable form on request?