Sampling AI-Drafted Patient Messages for Clinical Quality Review
Define a written sampling frame (which drafts, which period, which system), assign every draft to exactly one stratum by clinical risk, oversample high-risk strata, add risk-triggered review for specific signals, and record each cycle so it can be reproduced. Sampling measures quality. It does not replace human approval before a message is sent, and it does not act on each interaction the way runtime policy enforcement does.
Reviewer rubric and findings workflow
- 1
1. Freeze and draw
Export the frame, record the seed or start, apply strata, and draw the sample plus any triggered items.
- 2
2. Score against the rubric
Score clinical accuracy, completeness, appropriateness of tone, PHI handling, and whether the message needed clinician escalation instead of a reply.
- 3
3. Classify errors
Assign each finding to a defined error taxonomy and a severity rating, using definitions fixed before the cycle starts.
- 4
4. Escalate high severity
Route high-risk findings immediately to the responsible clinician and clinical safety lead, without waiting for the cycle report.
- 5
5. Assign disposition
Record the outcome for each item and the corrective action: prompt or policy change, clinician coaching, category restriction, or no action.
- 6
6. Verify the fix
Target the affected stratum in the next cycle to confirm the corrective action worked.
Evidence each review cycle should preserve
- The written protocol: frame definition, design, strata, sample sizes, oversampling rationale, and cadence
- The frozen frame export, random seed or start, and the list of items drawn
- Transaction log entries for each sampled draft: AI version, date and time, patient ID, responsible clinician
- Completed rubric scores, error classifications, and severity ratings with reviewer identity and date
- Dispositions, escalations, and corrective actions, with owners and verification results
- Reviewer training records and the policy on disclosing AI use to patients and clinicians
Why an informal subset gives weak assurance
Clinicians cannot review every AI-drafted portal or inbox reply for quality after the fact, and reviewing whatever is convenient produces results nobody can generalize from. If reviewers pick messages that look interesting, the findings describe the reviewer's attention, not the system's behavior. A defensible method starts from a written sampling frame and a stated selection procedure, so another reviewer could repeat the draw and get the same population.
The frame is the first design decision. Write down which messages are included (portal replies, inbox replies, or both), the review period, and the system of record from which the list is exported. Freeze the export for each cycle. If you use systematic selection, the frame must be ordered and numbered, and the random start must be recorded.
The frame should be built from the AI transaction log. ONC SAFER practices recommend maintaining and regularly reviewing a log of AI system use, similar to the EHR audit log, capturing the AI version, date and time of use, patient ID, and responsible clinical user. Those fields let each sampled item be tied to a specific draft and a specific accountable clinician.
Choosing a sampling design
CMS sampling guidance names simple random, systematic, stratified, and cluster designs, or combinations, as the most common, and expects the planner to identify the design used. Those documents come from Medicare program integrity work, so they define designs rather than set healthcare AI thresholds. The thresholds are your governance committee's decision, and the decision should be documented.
- Simple random: Draw a fixed number of drafts from the frame without replacement, with equal selection probability. Best for unbiased overall estimates. It under-represents rare, high-risk categories.
- Systematic: Number the ordered frame, pick a random start, select at a fixed interval. Easy to run and audit. Record the start and interval, and check that the ordering does not align with a pattern such as time of day.
- Stratified: Classify drafts into non-overlapping strata and sample within each. Supports oversampling where variation or risk is higher. Each draft needs exactly one stratum, such as a single primary message category.
- Risk-triggered: Pull drafts that meet defined signals, such as a clinical-urgency category, flagged PHI handling, or a clinician reporting a problem. Triggered items are reviewed on top of the random draw and reported separately, because they are not a probability sample.
Strata, sample sizes, and cadence
A practical protocol combines designs: stratified random sampling for measurement, plus risk-triggered review for known signals. Useful strata include message category (for example medication, symptoms, test results, scheduling, billing), presence of PHI, and clinician edit behavior (sent unchanged, lightly edited, substantially rewritten). Oversample strata where an error would cause the most harm. FDA/HHS survey guidance describes stratified designs with oversampling for cases where variation between strata is expected to be larger, and notes that probability sampling lets bias be quantified. That guidance addresses a non-AI survey context, so treat it as a rationale, not a threshold. Record the oversampling rate and the reason for each stratum.
Sample size follows from the question being asked. To estimate an error rate, size depends on the confidence level, the margin of error you will accept, and the rate you expect. To check that a rare error is not occurring, size the sample so that you are likely to see at least one case if the true rate equals your tolerance. As an illustration, if the tolerated rate is 1 percent and you want 95 percent probability of observing at least one error, the calculation is ln(0.05) divided by ln(0.99), or about 299 drafts in that stratum. Set these parameters per stratum, write them down, and revisit them after the first cycles.
Cadence should be a written policy. The January 2025 HIPAA proposed rule would require written policies for retaining and reviewing activity records, including review frequency, with each review documented. As of November 2026, that proposal should be treated as direction, not current law. Define cadence by risk: more frequent review for high-risk strata and after any model, prompt, or policy change, and a regular cycle for the rest.
Where runtime enforcement complements sampling
Sampling is retrospective and statistical. It tells you how often drafts fall short and where, but it cannot stop an individual draft from carrying a problem. SAFER practices state that AI-generated recommendations should be reviewed and approved by a responsible human before they are sent to patients, and they call for written authority for IT staff to disable the AI-enabled system at any hour if a problem occurs. Sampling supports neither of those controls. It measures how well the whole arrangement is working.
Policy enforcement in the execution path covers the other side. Trussed enforces policy at runtime, not after the fact, on every interaction: allow, redact, block, or escalate for human approval. Rules such as redacting PHI that should not appear in a draft, or escalating a message category to a clinician, act before a draft leaves the path. Enforcement also generates audit-ready evidence mapped to HIPAA, and it can be deployed self-managed in the customer cloud with configurable retention and data residency. Trussed runs in production with healthcare organizations.
Policy rules cannot fully assess clinical quality, such as whether a reply is clinically appropriate or complete. That is the job of sampled review. The two layers feed each other: sampled findings show which rules to add or tighten, and enforcement records show which strata carry the most intervention, which informs oversampling.
Two-part assurance model
Frame
A frozen, documented population of AI drafts for the review period.
Stratify and sample
Non-overlapping strata by category, PHI presence, and clinician edit behavior.
Review and act
Rubric scoring, severity, disposition, and corrective action.
Enforce at runtime
Policy applied to every interaction: allow, redact, block, or escalate.
Pair sampled review with runtime enforcement
Discuss how policy enforcement in the execution path and audit-ready evidence can support your clinical quality review protocol.
Request a DemoHow does your AI governance program compare?
See where your program has gaps in less than 2 minutes.