Is an LLM Clinical Documentation Tool High-Risk Under the EU AI Act?

Classification follows intended purpose

The EU AI Act bases risk classification on the intended purpose of the AI system, in line with existing EU product safety legislation. For a compliance team, this means "LLM clinical documentation tool" is not a classification. Two products built on the same model can land in different categories because one only transcribes and drafts notes while the other suggests what the clinician should do next.

An AI system can be high-risk in two ways. The first is being intended for a use case listed in Annex III. The second is being a safety component of a product covered by sectoral Union legislation, or being such a product itself. Systems subject to third-party conformity assessment under that sectoral legislation are always considered high-risk. For healthcare, the sectoral question is whether the tool qualifies under the Medical Device Regulation or the In Vitro Diagnostic Regulation, which is decided under those regulations and not under the AI Act.

The practical consequence is that the feature inventory and the documented intended purpose are the primary classification inputs. If marketing material, instructions for use and actual behavior disagree, the classification question stays open.

Where documentation ends and decision support begins

The dividing line is whether the output records what happened or influences what happens to the patient. The table below is an analytical reading of that line, not a classification ruling. Each feature must still be checked against the full Annex III text and the product pathway.

• Transcription of the encounter: Records what was said. Does not on its face match the health examples in official material. Lower relative risk.
• Drafting or summarizing visit notes for clinician review: Output is edited and signed by the clinician. Classification still depends on documented intended purpose.
• Suggesting diagnoses or treatments: Moves toward clinical decision support. Raises the medical device question and the Article 6(3) question about materially influencing decisions.
• Suggesting billing or coding: Not a listed health example. Assess against Annex III and the effect on decisions about individuals.
• Triage or prioritization of patients: Emergency healthcare patient triage systems are described as high-risk in recital 46. Strong high-risk indicator.
• Assessing eligibility for treatment: Systems that assess whether somebody can receive a certain medical treatment are listed as high-risk in Commission material.

Article 6(3) and the profiling caveat

Article 6(3) lets an Annex III system escape high-risk status where it does not pose a significant risk of harm to the health, safety or fundamental rights of natural persons, including by not materially influencing the outcome of decision making. The exemption is narrow in practice for clinical tools. A drafting assistant whose output is always reviewed and changed by a clinician has a stronger argument than a tool whose summary determines what a downstream reviewer sees.

The caveat matters as much as the exemption. An Annex III system that performs profiling of natural persons is always high-risk, whatever the Article 6(3) assessment concludes. Any feature that builds risk scores or patient categories from record data should be examined for profiling before the exemption is considered.

The exemption is not self-executing. The provider must document the assessment before market placement or putting into service. A deployer evaluating a vendor claim of non-high-risk status should ask for that dated assessment, including how profiling was ruled out.

Provider and deployer duties, and the regimes that apply anyway

If a documentation tool is high-risk, the requirements cover risk management, quality and relevance of data sets, technical documentation and record-keeping, transparency and information to deployers, human oversight, and robustness, accuracy and cybersecurity. Providers carry most of the design-time burden. Deployers must take appropriate technical and organisational measures to use the system per its instructions for use and monitor its operation on that basis. Where use per the instructions may present a risk under Article 79(1), the deployer must inform the provider or distributor and the market surveillance authority without undue delay and suspend use.

That last duty has an operational implication. A deployer cannot suspend out-of-scope behavior it cannot detect, so monitoring and the ability to halt a use need to exist before an incident, not after.

Classification does not switch other regimes on or off. For a covered entity or business associate, the HIPAA Security Rule technical safeguards apply whether or not the tool is high-risk under the AI Act, and GDPR obligations for health data apply separately in the same way. Treat the AI Act as an additional layer of evidence requirements, not a replacement.

When assessing a vendor, ask for five things: the documented intended purpose, the Article 6(3) assessment if claimed, the product-pathway position, the instructions for use supplied to deployers, and the record-keeping and human-oversight evidence each party can produce.

Which obligations call for runtime enforcement and evidence

Several obligations are about what happens during operation, so they need controls in the execution path and retained records. These include record-keeping, human oversight, keeping a tool within its intended purpose, and handling protected health information in prompts and outputs. A written policy is not evidence that these controls operated. Evidence has to show what the AI was allowed to do and what it did.

Trussed enforces policy in the execution path at the level of individual interactions and tool calls: allow, redact, block, or escalate for human approval. For a documentation tool, that can mean redacting PHI before it reaches a model, blocking a request that falls outside the declared scope, such as a triage query sent to a scribe, and routing a sensitive action to a human for approval. Each decision is enforced at runtime, not reviewed after the fact, and generates audit-ready evidence mapped to the EU AI Act and HIPAA, including 164.312(b) audit controls. Trussed runs in production with healthcare organizations and offers self-managed deployment in the customer cloud with configurable retention and data residency.

The limits are specific. Trussed does not determine whether a tool is high-risk, and it does not cover model provenance or lineage or end-to-end tracing of agentic workflows. Classification, the Article 6(3) assessment, data set quality, technical documentation and MDR or IVDR conformity depend on other controls and on the provider.