Best Practices Guide for Healthcare IT

    AI Metrics for a Health System CIO Monthly Review

    A health system CIO should review a fixed monthly AI scorecard in six categories: usage and adoption, spend and consumption, data and PHI exposure, policy enforcement outcomes, human-escalation volume, and audit evidence completeness. Each metric needs a defined data source, a named owner and a standing review question. Metrics drawn from enforcement in the execution path are more reliable than those reconstructed from billing dashboards or after-the-fact review.

    Can your tooling produce these metrics from runtime evidence?

    • Does it record what AI was permitted to do and what it actually did, not only usage and billing totals?
    • Can it report per model, agent, tool and coding assistant?
    • Can it show which systems, including software programs and agents, reach ePHI and under which access rights?
    • Are enforcement decisions made and recorded in the execution path, rather than reviewed afterward?
    • Can it report PHI-handling events, policy outcomes and escalations monthly, with a stated measure of evidence coverage?
    • Can retention and data residency be configured to your requirements?

    Treat the monthly review as an operating cadence

    A monthly AI scorecard is a fixed set of metrics, each with a named owner, a defined data source and a standing review question. Its purpose is to let the CIO answer five questions every month. Which AI systems are in use? Who is using them? What data do they touch, including PHI? How are consumption and spend trending? Is policy being enforced when AI acts?

    The inventory has to cover clinical, operational and administrative AI. ASTP/ONC data shows 71% of hospitals used predictive AI integrated with the EHR in 2024, up from 66% in 2023, with simplifying billing and facilitating scheduling among the fastest growing uses. Treat those figures as context, not as a target for your own metrics.

    The regulatory anchor is the HIPAA Security Rule. Section 164.312(b) requires hardware, software and/or procedural mechanisms that record and examine activity in information systems that contain or use ePHI. The access control standard limits access to persons or software programs that have been granted access rights, and the integrity standard requires protection of ePHI from improper alteration or destruction. Information System Activity Review is a Required implementation specification. The regulation does not name AI or prescribe metrics, so mapping a scorecard to these provisions is interpretation, and your compliance team should confirm it.

    HHS has proposed Security Rule changes that would require written policies and procedures that are reviewed, tested and updated regularly. As of November 2026 that rule is a proposal only. Documenting your review cadence in writing is sound practice either way.

    Usage, adoption and spend metrics

    These three metrics establish what exists and what it costs. The spend metric is a practitioner recommendation, framed here as visibility and accountability rather than cost reduction.

    AI inventory coverage measures the count of AI systems in use, including vendor-embedded features, models, agents and coding assistants, and the share that are sanctioned and assessed. The data source is usage records captured where AI is invoked, reconciled against procurement and contract records. The review question is what is in use that nobody approved. A signal of concern is sustained month-over-month growth in unsanctioned tool use.

    Active usage by function measures users and interactions per AI system, grouped by clinical, operational and administrative area. The data source is usage records tied to identity. The review question is whether usage matches the approved purpose. Concerning signals include a system used heavily outside its intended function, or usage that cannot be tied to an identifiable person or software program.

    Spend and consumption measures attributable spend and usage volume per system, with an accountable owner for each. Sources are vendor invoices and model usage records. The review question is whether every dollar maps to a known system and owner. Signals of concern are spend that cannot be attributed, and consumption that diverges from invoiced amounts. Billing dashboards show totals. They do not show what AI was permitted to do or what it did, so spend alone is not evidence of control.

    PHI exposure, enforcement, escalation and audit evidence

    The second group of metrics shows whether policy holds when AI acts. These are the metrics most dependent on instrumentation in the execution path.

    Data and PHI exposure counts PHI-handling events by AI system, and which systems reach ePHI under which access rights, including agents and tools. Source: runtime events recorded at the point of interaction. Review question: which systems touched PHI, and were they permitted to? Concern: repeated redaction or block events involving PHI from the same system or team. Public AI tools deserve particular attention. CMS guidance for its own users says never to input PHI into publicly accessible AI platforms, and that is a useful reference when writing your own rule.

    Policy enforcement outcomes counts allow, redact, block and rate-limit decisions per policy and system. Source: enforcement decisions recorded when they are made. Review question: is policy being applied, and where does it fire most? A rising block count may mean a policy gap or a misused tool. A falling count may mean better behavior or lost coverage, so read it alongside the evidence metric below.

    Human-escalation volume measures actions routed for human approval, time to resolution and outcomes. Source: the approval queue. Review question: are reviewers keeping up? Rising volume with growing backlog suggests policy is too broad or staffing is too thin. Treat this as a practitioner metric with no external benchmark.

    Audit evidence completeness is the share of AI activity involving ePHI that has a reviewable record of what was permitted and what happened. It ties to 164.312(b). Gaps here mean other metrics rest on partial data.

    A governance-completion metric is a useful addition. HHS's AI strategy offers a precedent in its governance throughput idea: the percent of high-impact use cases with completed assessments and independent reviews.

    Reading trends and assigning ownership

    No external source supplies thresholds for these metrics, so set your own. Run two or three months to establish a baseline, then review direction and concentration rather than single values. A flat total can hide a single system generating most PHI events.

    Pair metrics so each checks the other. Enforcement counts mean little without evidence completeness. Spend growth means little without inventory coverage. Escalation volume means little without resolution time.

    Assign each metric one accountable owner. ASTP/ONC data notes that hospitals use multidisciplinary governance teams and ask who is accountable for evaluating models. Apply the same question to each line of the scorecard, across clinical, operational and administrative AI. Record decisions and follow-ups from each review in writing so the cadence itself is auditable.

    Where Trussed fits

    Trussed enforces policy in the execution path across models, agents, tools, MCP servers and coding assistants. Each interaction or tool call can be allowed, redacted, blocked, rate limited or escalated for human approval, enforced at runtime, not reviewed after the fact. That enforcement generates continuous audit-ready evidence, mapped to HIPAA among other frameworks, which can feed the enforcement, escalation and evidence completeness metrics above.

    Trussed deploys self-managed in the customer cloud, with configurable retention and data residency. It runs in production with healthcare and pharma organizations, has 18+ months of continuous production in a regulated enterprise, and operates 100+ models under one policy plane.

    Monthly AI scorecard categories

    Usage and adoption

    Which AI systems are in use, by whom, and whether they are sanctioned.

    Spend and consumption

    Attributable spend and usage trends per system and owner.

    Data and PHI exposure

    What data AI touched, including PHI-handling events.

    Policy enforcement outcomes

    Allow, redact, block and rate limit results at runtime.

    Human-escalation volume

    Actions routed for human approval and how they resolved.

    Audit evidence completeness

    Share of AI activity with reviewable records.

    Build the scorecard on enforcement evidence

    See how policy enforced in the execution path produces the audit-ready evidence behind each monthly metric.

    See How Evidence Is Produced
    Next step

    How does your AI governance program compare?

    See where your program has gaps in less than 2 minutes.