AI Metrics for the Health System CIO: A Runtime-Evidence Framework

Why adoption and productivity metrics are not enough

AI in health systems is already spread across clinical and administrative work. ASTP/ONC survey data shows 71% of hospitals used predictive AI within the EHR in 2024, up from 66% in 2023. The most common uses were predicting health trajectories or risks for inpatients and identifying high-risk outpatients for follow-up. The fastest-growing uses were simplifying billing and facilitating scheduling. These figures cover predictive AI only, but they show the pattern a CIO has to report on: clinical and administrative uses growing side by side, owned by different leaders.

When each department defines its own metrics, the results lean toward adoption and productivity. Those numbers say little about what an AI system was permitted to do, what PHI it touched, or which controls applied. The NIST AI RMF treats measurement as part of risk management. It says AI systems should be tested before deployment and regularly in operation, and that measuring risk includes tracking metrics for trustworthy characteristics and human-AI configurations. Its MEASURE subcategories point to metrics with stated appropriateness and performance or assurance criteria. A metric with no criterion and no owner does not meet that intent.

The HIPAA Security Rule adds an evidence requirement. As of November 2026, 45 CFR 164.312(b) requires covered entities and business associates to implement hardware, software, and/or procedural mechanisms that record and examine activity in information systems that contain or use ePHI. The text does not mention AI. A reasonable reading is that an audit metric needs both a recording step and an examination step, so a count of AI interactions alone does not show that records were reviewed. The access control standard refers to access by persons or software programs, so AI tools that reach ePHI can reasonably be treated as in scope for access-rights metrics. Confirm that interpretation with counsel.

A tiered metric set

Define the five tiers once at system level, then let departments add measures beneath them. Keep PHI-handling and enforcement metrics separable from adoption and productivity metrics in the data model, because confidentiality and integrity of ePHI are Security Rule concerns and should not be averaged into a general performance score.

• Adoption and utilization: Inventory of AI systems by department, active users, interaction volume, and the share of AI systems with documented, granted access rights.
• Outcome and operational performance: Department-owned measures such as documentation turnaround, risk-prediction performance, billing throughput or scheduling fill rates, each with a stated criterion.
• Data protection and PHI handling: Interactions involving PHI, PHI redaction events, and blocked attempts to send PHI to unapproved destinations.
• Policy enforcement and exceptions: Allow, redact, block and human-escalation rates, plus exception count, approver and duration.
• Audit readiness: Evidence completeness, retention coverage, and time to produce a record for a given AI interaction.

Clinical and administrative departments differ in risk, not in definitions

Shared metrics should use identical definitions across the system: interaction volume, PHI-touching interactions, enforcement action rates, exceptions and evidence completeness. This is what makes departments comparable. A block rate in revenue cycle and a block rate in clinical documentation should mean the same thing, even when the expected values differ.

Department-specific metrics sit on top. Clinical uses such as documentation, decision support, imaging and patient communication carry direct patient impact and heavy PHI exposure. For these, add human-escalation and override rates, since the HHS AI Strategy describes AI as augmenting clinicians rather than replacing human decision-making, and add outcome measures owned by clinical leaders. The ASTP/ONC data notes that predictive AI used to monitor health and recommend treatments remains relatively low, possibly because of the higher risk of errors. That supports tighter reporting on clinical decision support.

Administrative uses such as revenue cycle, scheduling, supply chain, HR and IT usually weigh throughput and accuracy more heavily. Data sensitivity still varies. Revenue cycle and scheduling routinely involve PHI, while HR and IT can involve employee data and system credentials. Report PHI-touching interactions for every department rather than assuming administrative means low risk.

Metrics that can be sourced from runtime enforcement evidence

Some metrics describe what AI was allowed to do and what it did. These can come from records generated at the point where policy is applied to an individual interaction or tool call: the allow, redact, block and human-escalation rates, PHI redaction events, policy exceptions and evidence completeness. They are distinct from outcome metrics, which department leaders own and which cannot be derived from enforcement records.

Each needs a written definition and a data source before it is reported. Evidence completeness, for example, can be defined as the share of in-scope AI interactions that have a complete, retrievable record. Exceptions should record who approved them, for what scope, and for how long they stayed open. Rates need denominators and context. A rising block rate might mean policy is working or that a department is hitting a workflow problem, so read it with the department owner rather than treating it as a score.

Be explicit about scope. These measurements describe individual interactions and tool calls. They do not describe model lineage or end-to-end tracing of multi-step workflows, so do not present them as if they do. Board materials should state what each metric covers and what it does not.

Ownership, cadence and tooling evaluation

Assign every metric a named owner. Control and evidence metrics fit the CIO and compliance. Clinical outcome metrics fit clinical leaders, and administrative outcomes fit their department heads. Keep these groups separate in board reporting so that productivity gains are not read as proof that controls are effective. NIST AI RMF GOVERN applies across the risk management process, so record ownership, escalation thresholds and exception approval in governance documents, not only in dashboards.

Cadence is an organizational decision. A common starting point is monthly operational review of enforcement and exception metrics with compliance, and quarterly summaries for the board and clinical leadership. Test the metric pipeline itself on a schedule, checking that definitions are applied consistently and that records are complete.

To evaluate whether current tooling can support this framework, ask the questions below. Have legal and compliance confirm how 164.312(b) applies to each AI tool and vendor, including business associates, before any report describes a system as audit ready. HHS also proposed Security Rule revisions in January 2025, so track that rulemaking for changes to evidence expectations.

• Does the tool record, for each AI interaction touching ePHI, what access was granted, what was permitted and what happened, in a form that can be examined and exported?
• Which metrics come from system-generated records, and which from self-reported or manual input?
• Can metrics be broken out by department and by clinical versus administrative use case under one shared definition?
• How are exceptions, approvers and durations recorded, and how is evidence completeness measured?
• How long are records retained, and can they be mapped to NIST AI RMF MEASURE outcomes and 164.312(b) in an audit response?

Where runtime policy enforcement fits

Enforcing policy in the execution path is one way to produce the control and provenance metrics above. When enforcement happens at runtime rather than being reviewed after the fact, each allow, redact, block or escalation decision generates a record as a by-product, which gives the CIO system-generated evidence instead of department-reported figures.

Trussed enforces policy in the execution path across models, agents and tools, and produces audit-ready evidence mapped to HIPAA and NIST. It runs in production with healthcare organizations. Its scope is individual interactions and tool calls, so it supplies the enforcement and evidence tiers of this framework, not the outcome metrics that clinical and administrative leaders own.