Implementation Guide | Healthcare

    How to Measure How Much Clinicians Edit AI-Drafted Patient Messages Before Sending

    Link three records under one message identifier: the AI draft, the clinician's final sent version, and the send event with reviewer action and timestamps. Then compute a small, written set of metrics from them: send-without-edit rate, edit rate, normalized edit distance, semantic change and categorized edit types. Treat the results as evidence that a review step occurred, not as proof of draft quality or of adequate review. Handle the stored text as PHI throughout.

    Edit measurement is a definition and data capture problem

    Health systems that deploy generative AI to draft portal replies and inbox messages eventually face the same question from risk and compliance reviewers: did a clinician actually review this, and how much did they change? Answering it requires two things that are often missing. The first is a written definition of each metric. The second is a record that ties the draft, the final message and the send event together.

    Without a definition, teams report an edit rate that means different things in different dashboards. One counts any keystroke, another ignores whitespace, a third excludes the signature template. Without linkage, the numbers cannot be audited. A reviewer cannot trace a reported figure back to a specific draft, a specific responsible clinician and a specific send.

    Federal health IT guidance gives a useful frame. ONC's SAFER guidance says AI-generated recommendations should be reviewed and approved by humans who take responsibility for them before they reach patients. It also names responding to patient messages as an example of AI use in patient care. Edit measurement is one way to produce evidence that this review step is operating, which is a narrower goal than proving the messages are good.

    Candidate metrics and their failure modes

    No single metric is sufficient. Each answers a different question and fails in a predictable way. Define the normalization rules first (whitespace, line breaks, auto-inserted signatures and templates) so that formatting changes do not inflate or hide edits.

    • Send-without-edit rate: Share of drafts whose normalized final text equals the draft. Easy to compute and explain. It cannot distinguish a careful read followed by approval from a rubber stamp.
    • Edit rate: Share of drafts with any change beyond normalization. Sensitive to trivial changes such as a greeting or punctuation, so it overstates meaningful review.
    • Normalized edit distance: Minimum character or token insertions, deletions and substitutions to turn the draft into the final text, divided by the longer length. It measures volume of change, not significance. A one-word change to a dose or a negation can matter more than a rewritten paragraph.
    • Semantic change: Distance between embeddings of the draft and final text, or a model-based judgment of meaning change. It captures rewording that lexical methods overcount, but it depends on the embedding model, can miss small high-stakes changes, and needs version control of its own.
    • Categorized edit types: Each edit labeled against a rubric, for example tone, factual correction, added clinical content or removed content. It is the closest to clinical significance, but it requires a defined taxonomy, labeling effort and a plan for rater agreement.
    • Time-to-send: Interval between draft presentation and send, taken from system timestamps. Audit logs are described as a reliable source of clinician time data. The interval does not show that the draft was read, and it is confounded by interruptions.

    Data to capture at draft, review and send

    Start from the baseline that SAFER guidance describes for an AI transaction log: AI version, date and time of use, patient identifier and responsible clinical user. That baseline does not hold draft text, final text or reviewer action, so extend it. This extension is a design choice, not a stated requirement, and it should be written into policy before launch.

    Capture at the point of send rather than reconstructing later. The final text at send is the authoritative version, and reconstructing it from later chart state risks mismatches if the message is amended afterward. Use one message identifier across all records and write them append-only so that a modification would be detectable.

    • Draft record: draft text or a protected reference to it, draft version, AI version in use, generation timestamp.
    • Linkage keys: shared message identifier, patient identifier and thread identifier.
    • Reviewer identity: the responsible clinical user, and the role of any delegate who edited or sent.
    • Reviewer action: sent as drafted, edited then sent, discarded, regenerated or escalated.
    • Final record: final sent text or protected reference, send timestamp, and review-open timestamp.
    • Metric context: normalization rules version and metric definitions version used when scores were computed.

    Handling PHI in the measurement pipeline

    Drafts and final messages are PHI. The measurement pipeline therefore inherits the same access, retention and audit expectations as the clinical record. Decide early whether metrics can be computed inside the system that already holds the text, so that only scores, identifiers and timestamps leave it. Often the analytics team needs distributions and trends, not message content.

    Where text must be retained for audit, restrict query access by role, record who accessed it, and set a retention period that policy defines rather than leaving it to storage defaults. Where text is used for categorized edit review, sample rather than copy the full population, and apply the same least-privilege rules to the reviewers. Keep derived metrics separate from raw text so most users never need the latter.

    HIPAA's technical safeguards in 45 CFR 164.312 include an addressable integrity control against improper, undetected modification of electronic PHI in transmission. Whether and how that and the audit control expectations apply to stored draft and final pairs is a question for your privacy and security officers and counsel. A proposed Security Rule change would require written policies for retaining and reviewing activity records and periodic testing of technical controls. As of November 2026 it is a proposal only, but a documented review cadence for the measurement controls is sensible regardless.

    Interpreting results and their limits

    A low edit rate has several explanations. The drafts may be good. The clinician may be under time pressure. The clinician may defer to the draft because it looks fluent, which is a form of automation bias. The metric cannot tell these apart. A high edit rate is similarly ambiguous: it may signal drift in draft quality, a change in the prompt or model version, or personal style preferences.

    Use the metrics as screening signals. Segment by AI version, message type, clinician and time period, and watch for shifts after a version change. Pair them with workflow measures that SAFER guidance already recommends monitoring, such as unacknowledged inbox messages, messages sent through the portal and response time frames. ASPE has proposed audit-log indicators for in-basket burden, including authored messages opened for under one second. These could flag possible non-review, but they were offered as burden measures and would need validation for this purpose.

    Do not cite the fact that clinicians edit drafts as evidence that adequate review occurs. Testimony that edits happen is not measurement. Report metrics with their definitions and sample sizes, and avoid presenting edit volume as a quality score or a productivity claim.

    Where enforcement-generated evidence fits

    The EHR-side workflow holds the clinician's edits and the send action. The AI interaction layer holds a different, complementary record: what the model was asked, what it drafted, and which controls applied. Trussed enforces policy in the execution path on AI interactions, with allow, redact, block, rate limit and escalate for human approval. Enforcement at runtime, not review after the fact, produces audit-ready evidence of what the AI drafted and which policy decision applied, mapped to regimes including HIPAA and NIST. Trussed runs in production with healthcare organizations.

    That evidence can feed edit-measurement analysis by supplying the authoritative draft record, the AI version context and the control decision, such as whether PHI was redacted or the interaction was escalated for human approval. It does not measure clinical quality, and it does not replace EHR-side analytics that capture the final sent text and clinician activity. The two records are joined on the shared message identifier.

    Capture points for edit measurement

    Draft generated

    Draft text, AI version, timestamp, patient and message identifiers.

    Clinician review

    Responsible clinical user, review events and edit activity.

    Send

    Final sent text, reviewer action and send timestamp.

    Measurement

    Metrics computed from the linked draft and final pair.

    Plan the evidence behind your review metrics

    Talk through capture points, PHI handling and how enforcement evidence at the AI interaction layer can support audit-ready records of drafting and review.

    Talk to an Expert
    Next step

    How does your AI governance program compare?

    See where your program has gaps in less than 2 minutes.