What Regulators and Accreditors Expect from a Health System-Wide AI Operating Model

The expectations that apply, in three tiers

The expectations on a health system's AI use come from sources with very different weight. Treating them as one list leads to over-building in some places and gaps in others. This guide covers the HHS-level sources: HIPAA, ONC and the HHS AI Strategy.

The first tier is enforceable. The HIPAA Security Rule technical safeguards at 45 CFR 164.312 are in force as of November 2026. They contain five standards: access control, audit controls, integrity, person or entity authentication, and transmission security. Section 164.312(b) requires hardware, software, and/or procedural mechanisms that record and examine activity in information systems that contain or use electronic protected health information.

The second tier is a proposal. In January 2025, HHS published a Security Rule notice of proposed rulemaking. It describes audit controls as crucial for determining whether a security violation occurred, and it says OCR's enforcement experience suggests compliance with the audit-controls standard could improve with more specificity. Treat its proposed requirements as proposals, not obligations. It still shows where OCR attention is likely to go.

The third tier is direction and developer duty. ONC's HTI-1 final rule sets requirements for AI and machine learning based predictive decision support in certified health IT. The disclosure duties fall on EHR and certified health IT developers, who must describe how an algorithm was designed, developed and trained and give clinical users information to assess fairness, appropriateness, validity, effectiveness and safety. Health systems receive that information as users. The HHS AI Strategy emphasizes clear roles, comprehensive inventories of AI use cases, transparent risk-management practices, and policies and oversight mechanisms. HHS's December 2025 request for information adds a stated aim of a regulatory posture that is predictable and proportionate to risk. Neither document creates a binding duty for health systems, but both signal what a coherent model looks like.

Mapping expectations to operating model components

Each expectation implies a component of the operating model. The more useful question for a compliance leader is which kind of mechanism can satisfy it. Some can be enforced at the point of execution. Others depend on people, process and vendors.

• Accountability, roles and inventory: Signal: HHS AI Strategy. Met by organizational governance. Named owners and a complete inventory across clinical, operational and administrative uses cannot be produced by a technical control, though runtime records can help check the inventory against actual activity.
• Policy: Signal: HHS AI Strategy. The organization writes and approves policy. Runtime enforcement can apply it to each interaction and tool call.
• PHI protection and access: Basis: 164.312 access control, which limits access to persons or software programs granted access rights. Met at runtime by checking scope at execution and by allowing, redacting or blocking.
• Human oversight: Signal: HHS AI Strategy. Governance decides who approves and when. Runtime escalation routes specific interactions to a human before they proceed.
• Audit evidence: Basis: 164.312(b). Met by mechanisms that record and examine activity. Runtime enforcement produces a per-interaction record as a by-product of applying policy.
• Fairness, validity and safety assessment: Basis: HTI-1 developer disclosures. Met by vendor disclosure and the health system's own clinical assessment. Runtime enforcement does not establish it.

Applying 164.312(b) to AI models, agents and tool calls

The audit-controls text is technology-neutral. It allows hardware, software and procedural mechanisms and does not prescribe a tool or log format. Applying it to AI is an interpretation: a model, agent or tool call that uses ePHI is part of an information system that "contains or uses" ePHI, so activity there falls within the standard. Likewise, reading the access control standard's phrase "persons or software programs" to cover AI agents is an interpretation, though a reasonable one for a compliance team to adopt.

Two design implications follow. First, the standard says record and examine. A store of raw records that nobody can query or review is a weak answer. Records should be organized so that a reviewer can pick one interaction and see what policy applied and what occurred. Second, per-interaction and per-tool-call granularity is the most defensible reading of "activity," although the text does not state it as a requirement.

The NPRM also says a regulated entity should choose audit controls based on its risk analysis and organizational factors. A reviewer is therefore likely to read the risk analysis alongside the records. If AI use is absent from the risk analysis, even excellent records will leave a question unanswered. Access rights for AI agents and integrations should flow from the same access-rights process that the access control standard already references.

What runtime enforcement and evidence can cover

Runtime policy enforcement means the decision about what an AI system may do is made in the execution path, at the moment of each interaction or tool call, not reviewed after the fact. Trussed enforces policy in the execution path across models, agents, tools, MCP servers, AI sites and coding assistants. The actions are allow, redact, block, rate limit and escalate for human approval, applied to individual interactions and tool calls.

This maps to three expectations. PHI protection: a policy can redact PHI from a prompt or block a tool call that falls outside an agent's granted scope. Per-interaction oversight: a policy can hold a defined class of action for human approval. Evidence: because enforcement makes the decision, the same step produces the record of what policy applied and what action was taken. Trussed generates this audit-ready evidence continuously and maps it to HIPAA, among other regimes.

Deployment matters to a health system. Trussed can be self-managed in the customer's cloud, with configurable retention and data residency, so evidence containing sensitive context stays under the organization's control. Trussed runs in production with healthcare and pharma organizations, has run continuously in a regulated enterprise for more than 18 months, and generates over 30 GB of audit evidence per day. That volume is a reminder to plan retention and review capacity, since evidence only helps if someone can examine it.

What stays outside runtime control

A restrained account of runtime enforcement has to include its limits. Accountability, roles, committee decisions and stakeholder engagement are organizational governance, and no technical control substitutes for them. The same is true of the inventory: someone must decide what counts as an AI use case and own each one.

Clinical validation is also outside scope. Whether a predictive tool is fair, valid, effective and safe depends on the developer's disclosures and the health system's own assessment. Model-level lineage and training-data transparency are developer responsibilities under HTI-1. Runtime enforcement does not produce them, and Trussed does not provide model or dataset provenance. Finally, HTI-1 attaches to certified health IT. Where AI is deployed outside it, the health system's own policy has to carry the expectation.

The practical result is a division of labor. Governance defines what AI should be allowed to do and who answers for it. Runtime enforcement makes that definition operate on each interaction and keeps the proof. Weakness on either side shows up when a reviewer asks for both.