AI Governance for Medical Device Software

Runtime control points in SaMD and AIaMD stacks

Modern stacks often combine deterministic device software with probabilistic models, agent or orchestration layers, external tools or APIs, and regulated data stores. That expands the control surface beyond classical embedded firmware. Governance is effective when authorization sits in the inference and orchestration path before side effects occur.

1. 1

   Identity

   Distinct machine identities for AI agents across training, validation, and production.

2. 2

   Permissions

   Default-deny, operation-level least privilege for data paths and tool actions.

3. 3

   Policy decisions

   Pre-effect authorization of model use, context retrieval, and tool calls.

4. 4

   Audit evidence

   Correlated logs of identity, policy version, model/version, action, and outcome.

Runtime control surface

These four layers form the practical enforcement surface for medical device AI at runtime. Each layer should be explicit, versioned, and testable within the existing software lifecycle.

What governance obligations apply to AI-enabled medical device software

Obligations for AI-enabled medical device software follow from intended use, residual risk, quality system controls, and applicable change processes. Runtime governance does not replace those obligations. It makes them enforceable when models, agents, data access, and tools act in production.

Teams need clear ownership so that quality and regulatory accountability for AI runtime controls stays with the appropriate functions. Platform teams implement enforcement; they do not redefine residual risk or intended use.

Implement identity, least privilege, and tool-call governance

Provision, rotate, and separate AI agent identities, secrets, and permissions across training, validation, and production. Apply default-deny authorization to model selection, regulated data access, and tool calls so unapproved actions cannot proceed.

Prefer allowlists over discovery. Production agents should not dynamically attach unreviewed tools, models, or data sources that expand clinical or privacy risk without change control. Bind configuration together so release baselines join model artifacts, policies, prompts, tool schemas, and device software versions with provenance suitable for configuration management and SBOM practices.

Change control, monitoring, and evidence that survive updates

Prompt, tool schema, policy, and model changes should enter IEC 62304 and ISO 14971 change control, and PCCP processes when applicable. Audit fields must reconstruct who or what acted, under which policy and software version, with model, tool, and disposition context.

Monitoring and rollback evidence should demonstrate continued control effectiveness after model or software updates. Human oversight roles, escalation paths, and change approvers must be defined and testable for high-risk actions. Test degraded modes so denial, rollback, and offline states fail closed rather than widening permissions or skipping oversight gates.

Design choices that keep governance inspectable

Implementation checklist for medical device AI runtime governance

• Runtime enforcement points exist for model selection, regulated data access, and tool calls, and unapproved actions are default-deny.
• AI agent identities, secrets, and permissions are provisioned, rotated, and separated across training, validation, and production.
• Audit fields reconstruct who or what acted, under which policy and software version, with model, tool, and disposition context.
• Prompt, tool schema, policy, and model changes enter IEC 62304 and ISO 14971 change control, and PCCP processes when applicable.
• Monitoring and rollback evidence demonstrate continued control effectiveness after model or software updates.
• Human oversight roles, escalation paths, and change approvers are defined and testable for high-risk actions.