Emergency Department AI Triage Governance: Runtime Controls

What Runtime Governance Means for AI Triage Agents

Emergency department AI triage tools are increasingly positioned as decision-support layers over the Emergency Severity Index (ESI), the five-level triage algorithm most U.S. EDs already use. As these tools move from producing a recommended acuity level to actively calling EHR, bed management, lab ordering, or alerting systems, they begin to function as autonomous or semi-autonomous agents rather than passive scoring tools.

Runtime governance addresses this shift directly. It is the layer of controls enforced at the moment an agent attempts an action, distinct from pre-deployment model validation such as FDA's review process for AI-enabled device software. A model can be clinically validated and still operate without any enforced boundary on what systems it is permitted to touch, what data it can write, or what record exists of what it did. Runtime governance closes that gap by governing agent identity, permission scope, policy enforcement, and audit logging as separate, ongoing controls.

Four Runtime Control Pillars

Effective runtime governance for ED AI triage rests on four complementary controls. Each addresses a different failure mode when an agent can initiate tool calls against clinical systems.

Agent Identity and Least-Privilege Permission Scoping

Identity and permission design determine whether triage-related actions can be attributed, constrained, and reviewed. The following sequence is a practical baseline for ED deployments.

1. Distinct agent identity

   Authenticate the agent separately from clinician credentials so every tool call can be attributed to the agent itself.

2. Per-system permission scopes

   Define separate least-privilege boundaries for EHR, bed management, lab, and alerting access rather than a single broad role.

3. Policy enforcement point

   Place a control layer between the agent and each system that evaluates calls before execution.

4. Cross-system identity federation

   Coordinate identity and permissions across separately governed vendors and IT departments that control each system.

Why This Layer Is Missing in Most Current Deployments

No dedicated regulatory or technical standard currently mandates runtime tool-call governance for ED AI triage agents. Existing frameworks address adjacent but different problems.

HHS ASTP/ONC's HTI-1 rule requires transparency disclosures about a predictive decision support intervention's data inputs and validation, but says nothing about what actions the tool is subsequently permitted to take. FDA's Predetermined Change Control Plan guidance governs how manufacturers document planned model changes, which is a pre-market and change-control concern, not a real-time enforcement mechanism.

The HIPAA Security Rule's audit control requirement (45 CFR 164.312(b)) establishes a baseline for recording system activity involving protected health information, but was not designed with agentic tool-call granularity in mind. The result is a governance gap: health systems deploying AI triage agents that call clinical systems are currently expected to adapt general frameworks, such as NIST's AI Risk Management Framework and NIST SP 800-53's access control family, rather than rely on a healthcare-specific runtime standard.

Real-Time Enforcement Versus Pre-Deployment Validation

OWASP's guidance on large language model applications identifies "Excessive Agency" as a distinct risk category, describing harm that results when an agent is granted more functionality, permissions, or autonomy than its task actually requires. This risk is not addressed by clinical validation of the underlying model.

A model can produce accurate acuity recommendations while the agent surrounding it retains unnecessary write access to a lab ordering system or an unrestricted ability to trigger alerts. Runtime policy enforcement is the mechanism that closes this gap: permission decisions evaluated at the moment of each tool call, rather than assumed to be safe because the model passed validation earlier in its lifecycle.

This distinction matters operationally as well as technically. Enforcement mechanisms in an ED context must operate with latency compatible with time-sensitive workflows, since a policy check that introduces meaningful delay to a triage-related action creates its own patient safety risk. Permission boundaries should also be reviewed and re-scoped as the agent gains new integrations, for example moving from read access to EHR records to write access for lab orders.

Governance Responsibility Across Vendors and Systems

HIPAA's audit control requirement is a compliance floor, not a governance ceiling. Logs generated to satisfy 45 CFR 164.312(b) can often be extended to capture AI-specific fields, but they were built for general system activity, not for the granularity needed to reconstruct why an autonomous agent took a specific triage-influencing action. Health systems deploying AI triage agents typically need supplementary logging design on top of existing HIPAA infrastructure.

A related and unresolved problem is ownership. When an AI triage agent interacts with an EHR from one vendor, a bed management system from another, and a lab information system from a third, governance responsibility for that agent's permissions can be unclear, since no single vendor typically controls the full permission chain.

Reconciling change-control processes for model updates, such as those addressed by FDA's PCCP guidance, with ongoing runtime permission reviews is a separate coordination task that current frameworks do not resolve. Health systems evaluating or operating AI triage agents should treat this as an open governance question requiring explicit internal ownership, not an assumption that vendor compliance covers it.

Audit Logging for Triage-Influencing Actions

Audit logs for AI triage agents should support reconstruction of who initiated an action, what was requested, whether policy allowed it, and what happened next. At minimum, capture:

• Agent identity for the authenticated principal that issued the call
• Requested action including target system and operation type
• Permission decision (allow or deny) and the policy rule applied
• Outcome of the call, including failures and partial writes
• Timestamp and correlation IDs sufficient to join events across systems

Assessing Whether Your Deployment Has Adequate Runtime Governance

Use the following checks when reviewing an existing or planned ED AI triage deployment:

• Distinct agent identity: Confirm the AI triage system operates under its own authenticated identity, separate from clinician credentials.
• Scoped tool-call permissions: Verify permission scope for each system the agent touches, limited to least-privilege access.
• Real-time enforcement: Confirm policy checks occur at the point of each tool call, not only through pre-deployment validation.
• Audit log completeness: Check that logs capture identity, action, decision, and outcome for every AI-initiated call.
• Permission review process: Confirm a defined process exists for re-scoping permissions as new integrations are added.