Healthcare AI Implementation Guide

    How to Prevent AI from Drafting Replies to Patient Messages That Report Emergency Symptoms

    Place a deterministic control point in the message-handling path that checks each inbound patient message for emergency-symptom content before any draft is requested or surfaced. When the check fires, or when it cannot complete, block drafting, mark the message urgent, route it to clinical staff, and record the decision as evidence. A prompt instruction alone cannot guarantee this outcome, because the model is the component being constrained.

    Route flagged messages to clinical staff

    1. 1

      1. Mark urgent

      Assign the highest urgency level in your existing message-priority scheme and present the message distinctly in the clinical inbox.

    2. 2

      2. Route to a monitored queue

      Send to a staffed triage queue, not to an individual who may be away. Include the detected category as context, not as a draft.

    3. 3

      3. Start an escalation timer

      If the message is unread or unanswered within the defined period, re-escalate to a second role or on-call contact.

    4. 4

      4. Clinician responds

      A clinician reviews the message and responds through the normal clinical process. No AI draft is offered.

    5. 5

      5. Record disposition

      Capture whether the flag was confirmed, the response time, and any re-escalation, for tuning and review.

    Evidence each decision should produce

    • Message and thread identifier, timestamp, and the pseudonymous patient or encounter reference
    • Detector name and version, result, detected category, and confidence or threshold applied
    • Policy identifier and version, and the action taken: allow, redact, block or escalate
    • Whether a model call was made, and confirmation that none was made when the action was block
    • Routing target, urgency level assigned, escalation timer outcome, and any re-escalation
    • Clinician acknowledgement, final disposition, and any manual override with the user and reason

    Why this is a runtime enforcement problem

    Messages that report chest pain, stroke signs, suicidal ideation or difficulty breathing need a human response path, not a drafted reply. A draft can delay escalation, reach the patient without clinician review, or anchor a busy clinician on a routine answer. Each of those outcomes is a failure of the workflow around the model, not only of the model.

    That is why the control belongs in the execution path. A prompt can tell a model to decline emergency messages, but the instruction is processed by the same probabilistic system it is meant to constrain. It also leaves no independent record of what was decided. A deterministic check outside the model can be tested, versioned, and shown to have run on every message.

    ONC SAFER guidance points the same way. It says AI-generated recommendations should be reviewed and approved by humans who take responsibility for them before they are sent to patients. It also says critical and time-sensitive messages should be clearly differentiated from routine ones. Together these imply that an emergency-flagged message should never reach a drafting step that could put unreviewed content in front of a patient.

    Three places to put the control

    The options differ in how much of the outcome you can verify. The comparison below is a design assessment, not a measured benchmark.

    • Inside the prompt: Easy to add and cheap to change. The model decides whether to comply, so behavior can vary and cannot be proven for a given message. It produces no decision record separate from the model output. Treat it as a supplement only.
    • Application-layer rules: Deterministic code in the portal or messaging service, such as keyword checks before the drafting call. Verifiable and testable. The risk is coverage: each drafting path, integration or new feature must carry its own copy of the rule, and evidence format varies by team.
    • Policy enforced in the execution path: One policy applies to every interaction that reaches a model, whichever application originated it. The decision is made before the model call, can fail closed, and generates evidence in a consistent shape. This needs a detection signal and a routing target to be useful.

    Define block and escalate conditions

    Start with the decision, then the signal. The policy needs a small set of explicit conditions and a defined result for each.

    Block drafting and escalate when the detector reports an emergency-symptom category above your threshold. Categories should map to your clinical triage categories, for example cardiac, neurological, respiratory and self-harm. Block and escalate also when the detector times out, errors, or is unavailable. Failing closed is the safer default for this use case, because an unavailable detector must not become a path to an unreviewed draft.

    Allow drafting only when the detector completes and reports no emergency content. Redaction has a narrower role here. It can strip unnecessary identifiers from content sent to a model for routine messages, but it does not make drafting acceptable for an emergency message.

    Apply the check at two points. The first is before the draft request. The second is before the draft is surfaced to a clinician, which catches threads that changed after the first check, such as a patient adding a follow-up message. Whether detection uses lexicons, a trained classifier, a model-based check, or a combination is a separate engineering choice. Whatever you select, set thresholds conservatively and validate them against your own labeled messages before launch.

    Provide a disable path for the whole drafting feature. If monitoring shows detection quality has degraded, operators should be able to stop drafting without a code release.

    • Block and escalate: Emergency category detected, or detector failed, timed out or was unavailable.
    • Allow: Detector completed with no emergency content, and the draft still goes to clinician review.
    • Redact: Remove identifiers not needed for a routine drafting request.
    • Disable: Operator switch that stops drafting across the feature.

    Handling false negatives and false positives

    No detector is perfect, so design for both error types instead of assuming either away.

    A false negative is an emergency message that passes the check. Reduce its impact by keeping clinician approval as the gate for every AI draft, by running the second check before surfacing, and by testing before deployment and monitoring in production. Review a sample of allowed messages on a schedule, and review every clinician report that a message should have been flagged. Track these as incidents, feed them into threshold and rule changes, and re-run your test set whenever the detector or the underlying model changes.

    A false positive is a routine message that gets blocked. The cost is clinician time and lost drafting convenience, not patient harm, so for this use case it is the more acceptable error. It still needs measurement. If staff learn that flags are mostly noise, they may stop treating them as urgent. Record the clinician's disposition of each flagged message so you can measure how often flags were confirmed and tune accordingly.

    Automation bias is a separate failure mode. FDA material on LLM risk management notes that human factors affect the safety of any human-machine team. Suppressing drafts for emergency messages removes the anchor entirely for the highest-risk cases, which is one reason to block rather than to show a draft with a warning label.

    Assign an owner for the detector, the thresholds and the review cadence. Without a named owner, tuning tends not to happen.

    Control point before drafting

    Inbound message

    Patient portal message is received and stored.

    Detection

    Emergency-symptom check runs before any draft request.

    Policy decision

    Allow drafting, or block and escalate to a human.

    Clinical routing

    Flagged message is marked urgent and sent to staff.

    Evidence

    Each decision produces a record for audit review.

    Enforce the block before a draft exists

    Talk through where your drafting pipeline needs a control point, which decisions to enforce, and what evidence your audit reviewers expect to see.

    Talk to an Expert
    Next step

    How does your AI governance program compare?

    See where your program has gaps in less than 2 minutes.