Measuring AI Standardization Maturity Across Health System Departments

What AI standardization maturity measures

AI standardization maturity describes how consistently a health system applies one set of AI rules across clinical, revenue cycle, IT, research, and administrative departments, and how reliably those rules are enforced and evidenced. It is not a measure of how much AI a department uses. It is a measure of how uniform and verifiable that use is.

The need is practical. ONC data show 71% of hospitals reported using predictive AI within the EHR in 2024, up from 66% in 2023. The same reporting shows that most hospitals evaluated at least some models for accuracy and bias, but fewer did so for all or most of their models. That gap between some and most is the pattern a maturity model should expose at department level.

A framing note. The NIST AI RMF is voluntary, and the HIPAA Security Rule amendments published as a proposal in January 2025 are proposed only as of November 2026. The model below is therefore a practitioner measurement framework, not a regulatory standard. It borrows from the NIST MEASURE function, which calls for selecting metrics starting with the most significant risks, documenting risks that will not or cannot be measured, and regularly reassessing whether metrics and existing controls are effective.

Staged maturity levels

Levels are defined by the strongest behavior a department can demonstrate consistently, not by intent. Score each dimension separately, then set the department level at its lowest dimension score. This weakest-link rule is a design choice. It can understate progress, but it prevents strong documentation from masking an unenforced PHI rule.

• Level 1: Ad hoc: Departments choose tools and approval paths independently. No shared inventory. PHI handling depends on individual judgment.
• Level 2: Documented: A common written policy exists and an inventory is started. Compliance relies on attestation and manual review.
• Level 3: Partially enforced: Policy is applied in the execution path for some tools or models. Others remain documented only. Evidence is assembled by hand.
• Level 4: Enforced and evidenced: One policy set is enforced across most inventoried use cases. Evidence is generated by enforcement and can be produced on request.
• Level 5: Measured and reviewed: Coverage and control effectiveness are measured on a schedule. Gaps trigger policy changes, and unmeasured risks are documented.

Finding the gap between written policy and enforced behavior

The most useful output of a first assessment is not the score. It is the difference between a department's documented level and its enforced level. A department can be Level 2 on paper for every dimension and Level 1 in practice if nothing in the execution path applies the policy.

Test each written rule by asking where it is applied. If the answer is a training module, a review meeting, or an intake form, the rule is documented. If the answer is a control that allows, redacts, blocks, or escalates a specific interaction or tool call before it completes, the rule is enforced. Both are legitimate, but they should be scored differently and reported separately.

Run the assessment in the same order for every department so results stay comparable. Clinical teams may need tighter escalation triggers, and revenue cycle or research teams may expose different PHI handling patterns, but the scoring method should not change.

• Build the AI use case inventory for each department, including coding assistants and agents.
• Map each written policy rule to the point where it is applied.
• Classify each rule as documented, partially enforced, or enforced in the execution path.
• Sample a period of activity and test whether evidence can be reconstructed.
• Record risks the department does not or cannot measure.
• Set a reassessment date and an owner for each gap.

Evidence an auditor would expect to see

For HIPAA purposes, the question behind AI evidence is the one auditors ask of any system touching electronic PHI: can the organization show what happened? The Security Rule's audit controls standard, at 164.312(b), is the usual reference point. The rule predates current AI tools and does not prescribe AI-specific controls, so the mapping from AI activity to audit controls is a judgment your compliance team must document.

In practice, strong evidence answers two questions for a given period. What was each AI system permitted to do under policy? What did it actually do, including what was redacted, blocked, or sent for human approval? Evidence that shows only the second question is activity history. Evidence that shows both lets an auditor test whether policy was operative.

Two further habits help. First, retain the reviewer decision for every escalation, since an escalation path without recorded outcomes cannot be scored above Level 3. Second, reassess control effectiveness on a schedule, as NIST MEASURE 1.2 describes. The proposed HIPAA amendments would require compensating controls to be reviewed, documented, and signed by the designated Security Official at least every 12 months. That requirement is proposed, not in force, but the cadence is a reasonable forward-looking practice.

What a runtime control and evidence layer must provide

Moving departments from Level 2 to Level 4 usually requires something that applies policy where AI actually executes. Documentation and review cannot raise enforcement coverage on their own. Evaluate any such layer against the same inventory used for scoring, not against the list of tools the vendor happens to cover.

Trussed is one example. It enforces policy in the execution path across models, agents, tools, MCP servers, AI sites, and coding assistants, with actions that include allow, redact, block, rate limit, and escalate for human approval. Its scope is individual interactions and tool calls. Enforcement generates audit-ready evidence mapped to HIPAA and NIST, and it can be deployed self-managed in the customer cloud with configurable retention and data residency. Trussed runs in production with healthcare organizations. It does not provide transaction-level governance, model or dataset lineage, or end-to-end tracing of agentic workflows, so those needs must be met another way.

Whatever product is under review, ask for documentation on each of the following.

• One policy set applied across models, agents, tools, and coding assistants in every department.
• A clear split between actions enforced in the path and actions only recorded.
• Technical PHI controls such as redaction and blocking, not instruction alone.
• Human escalation with named approvers and recorded decisions.
• Evidence per interaction, with a stated mapping to HIPAA Security Rule controls and NIST AI RMF functions.
• Explicit statements of what the layer does not cover.