Governance of Generative AI That Suggests Differential Diagnoses to Clinicians

Define intended use and keep the clinician in charge

Start with a written scope for the system. Define intended use and keep clinician decision authority explicit, so that the clinician remains the decision-maker for every suggestion the AI produces.

Apply least privilege to patient data and tools

Restrict data and tool access to least privilege. Set per-role, per-task limits on patient data and on the tools the AI can use.

Enforce policy at inference and tool-call time

Runtime policy means checks at inference and at each tool call. Governance applies while the system is running, not only when the model is first approved.

Audit evidence to capture for every suggestion

Log every suggestion with its inputs and versions. Decision logging gives you a record of what was suggested, to whom, and why. Capture the following for each suggestion:

• Request inputs and a reference to the patient and encounter context
• Retrieved sources and the data actually accessed
• Tool calls with parameters and returned results
• Model, prompt, and configuration versions in effect
• Suggestions displayed and the clinician who received them
• Clinician actions and overrides, plus the policy decisions applied

Control change and monitor production behavior

Change control

Model and prompt changes should be approved, versioned, and reversible.

Monitoring and escalation

Monitor production behavior with ongoing safety review, defined escalation, rollback, and named owners.

Evaluate platform and vendor capabilities

When evaluating a platform or vendor, assess each of the runtime control points for diagnostic-suggestion AI listed below.