Health AI Post-Market Surveillance Metrics: What to Monitor and How Often

Define post-market surveillance as a lifecycle control

Health AI monitoring after deployment should establish whether a system continues to perform safely, fairly, securely, and within its approved use. The monitoring plan should connect approved use, risk tier, ownership, operational thresholds, review cadence, escalation paths, and retained evidence.

Metrics to monitor after deployment

The following metric categories provide a practical baseline for deployed health AI systems. The exact measures should be appropriate to the task, deployment context, approved use, and risk level.

• Model performance and calibration: Track accuracy-related measures appropriate to the task, confidence or score distributions where available, calibration changes, failure rates, exception rates, and degradation from the deployment baseline.
• Data and population drift: Monitor input data quality, missingness, source changes, distribution shifts, device or location differences, coding changes, and population shifts that could make pre-deployment validation less representative.
• Fairness and subgroup performance: Where legally and operationally feasible, compare performance across relevant subgroups, sites, care settings, specialties, devices, or workflows. Aggregate performance can hide localized degradation.
• Safety, usability, and workflow behavior: Track adverse events, harmful outputs, user complaints, override rates, acceptance rates, abandoned outputs, human review outcomes, and cases where AI recommendations conflict with clinical judgment.
• Security, privacy, and policy compliance: Monitor unauthorized access attempts, privacy events, prompt or request patterns that violate policy, unsafe outputs, anomalous usage, latency, availability issues, and failed runtime controls.
• Agent and tool-call behavior: For AI agents, track tool calls, read and write actions, permission use, approval outcomes, agent-to-agent interactions, denied actions, and attempts to operate outside approved scope.

Set monitoring cadence by risk, not by habit

Monitoring cadence should be risk-based. Systems with higher clinical impact, frequent updates, changing patient populations, or access to enterprise tools require tighter runtime controls, more frequent review, and immediate escalation paths for safety, security, privacy, or unauthorized action events.

Example cadence framework

A practical cadence framework should consider clinical risk, workflow criticality, update frequency, patient population change, data-source change, and whether the system or AI agent has access to enterprise tools. Higher risk and higher agency should result in more frequent review and clearer escalation thresholds.

Runtime auditability for health AI monitoring

Post-market surveillance depends on evidence that can reconstruct what happened. For each meaningful AI interaction, the enterprise should capture enough runtime context to support investigation, audit, and corrective action. Useful records include the model or agent version, configuration, request metadata, input source, output, confidence or score where available, user identity or role, user action, tool calls, policy decision, exception, approval step, and escalation path.

For AI agents, runtime auditability should show not only what the model generated, but what the agent attempted to do. This includes which tool was called, what permission was used, whether the action was read-only or write-capable, whether human approval was required, and whether the action was allowed or denied by policy. This distinction is important because excessive agency is a recognized security concern for LLM-based applications that can call tools or take actions.

A practical architecture links the AI system inventory or model registry to the runtime control plane. The inventory should identify the approved use case, owner, risk tier, deployment environment, data sources, validation evidence, monitoring plan, and change history. Runtime controls should enforce least privilege, agent identity, approved tool access, and policy decisions. Logs should be tamper-resistant, searchable, retained according to policy, and exportable for internal audit, vendor review, regulatory inquiry, or incident reconstruction.

Trussed AI provides runtime governance and security capabilities for enterprise AI agents, including runtime monitoring, policy enforcement, agent identity, permissions, least privilege, tool approval workflows, and audit logging. In a health AI surveillance program, these controls can provide part of the evidence layer showing whether an AI agent stayed within its approved scope and how exceptions were handled.

Evidence to retain for governance and corrective action

Post-market surveillance should produce durable evidence that supports scheduled governance, incident response, corrective action, vendor review, and audit readiness.