Standardizing AI audit evidence across every department in a health system

Capture evidence where policy is enforced

Consistency is difficult when evidence comes from each tool's own log. Every vendor records different events in a different shape, and some record very little. Normalizing those logs after the fact is a documentation exercise that has to be repeated for each new tool.

A more durable design is to capture the record at one point in the AI execution path, where each request to a model and each tool call passes through a policy decision. At that point the actor, the policy applied and the decision are all known, and the record can use one schema regardless of department or model. This is a design inference rather than a regulatory requirement, and it has a clear limit: only traffic that passes through the control point is covered. Document which AI tools and agent paths sit outside it, and treat those as exceptions with their own evidence arrangements.

Protect the evidence store itself. The HIPAA integrity standard calls for protecting ePHI from improper alteration or destruction, and evidence that can be edited is hard to defend. Evidence that references PHI should be handled as sensitive data in its own right.

Map the record to HIPAA and related frameworks

Test the record against the 164.312(b) wording: does it let you both record and examine activity? A record that exists but cannot be searched by actor, system or data category supports recording but not examination. NIST SP 800-66 Rev. 2 includes a section on Audit Controls (164.312(b)) in its HIPAA Security Rule guidance, and it is a sensible reference for the review.

Watch the regulatory text. In January 2025 HHS proposed to redesignate the audit controls standard as 164.312(d)(1) and rename it audit trail and system log controls. As of November 2026, 164.312(b) remains the current citation, and the proposal should be tracked rather than cited as law.

If any department operates in or serves the EU, the EU AI Act requires high-risk AI systems to allow automatic recording of events over the system's lifetime, with traceability appropriate to the intended purpose. Legal should assess whether and when that applies. The wording also supports defining evidence per use case and risk tier rather than one maximal record for everything. For NIST AI RMF, map your fields to the framework's own text, and keep the mapping in the same register as the evidence schema so audits can follow it.

Runtime enforcement versus after-the-fact review

After-the-fact review starts from whatever each tool happened to record, then asks someone to reconstruct what policy applied. Gaps are found during the audit, when they can no longer be filled. Evidence produced by runtime enforcement is different: the policy decision is made in the execution path, and the record of that decision is generated by the same step. Allow, redact, block, rate limit and escalate each leave a recorded outcome, so the evidence shows both what was permitted and what happened.

Trussed is one approach to this. It enforces policy in the execution path across models, agents, tools, MCP servers, AI sites and coding assistants, at the level of individual interactions and tool calls, with escalation for human approval. Audit-ready evidence is generated by enforcement and mapped to HIPAA and NIST. It can be deployed self-managed in the customer cloud, with configurable retention and data residency. Trussed runs in production with healthcare and pharma organizations, including 18+ months of continuous production in a regulated enterprise.

Any control point you evaluate should be able to answer the same questions: does it produce one record per interaction and tool call, does it show the outcome of each decision, and which paths does it not cover?

Sequence the rollout across departments

Roll out by risk and PHI exposure rather than by org chart. The steps below keep the evidence standard stable while departments onboard one at a time.

• 1. Inventory AI use: For each department and tool, record the evidence that exists today, its format, retention and storage location.
• 2. Fix the schema: Agree the minimum record and test it against 164.312(b) and the NIST SP 800-66 Rev. 2 audit controls section.
• 3. Pilot one department: Choose a high-PHI-exposure area and run the control point there first, including escalation paths for human approval.
• 4. Record approvals: Capture approver identity and outcome as part of the same evidence record, not in a separate ticket system.
• 5. Validate completeness: Have internal audit sample records against the schema before the department is declared onboarded.
• 6. Expand by exposure: Move to revenue cycle, research, administrative, IT and patient-facing teams, recording uncovered tools as exceptions.

Decide ownership, retention and residency centrally

No regulation supplies a retention period for AI evidence, so retention is organizational policy. Set it centrally, document the reasoning, and allow documented per-department variations only where a use case demands them. The same applies to storage location and residency: decide where evidence may live, especially where it references PHI, and configure the control point to match.

Name one accountable owner for the evidence standard and a named owner in each department. ONC SAFER guidance notes that organizations using AI-enabled systems share responsibility for AI safety with developers and recommends local monitoring, which fits this model. Evidence ownership should include who reviews escalations, who responds to auditor requests, and who approves changes to the schema.

• One accountable owner for the standard, plus per-department owners
• Retention set as written policy, with exceptions documented
• Residency and storage location decided before onboarding
• Evidence store protected from alteration or destruction
• Internal audit able to test completeness independently
• Uncovered AI tools and agent paths listed and reviewed