Determining Whether an LLM Claims Summarization Copilot Is High-Risk Under the EU AI Act
Classification depends on the system's intended purpose and on how much its summaries influence decisions about individuals' access to essential services or benefits, not on the fact that it uses an LLM. This is not legal advice.
A structured classification test
The determination for a claims summarization copilot rests on four inputs. Each is covered in the sections that follow.
| Input | What it establishes |
|---|---|
| Intended purpose | What the system is designed and documented to do, and in which workflow. |
| Annex III fit | Whether that purpose falls within a listed high-risk category. |
| Decision influence | Whether summaries prepare a human decision or effectively determine it. |
| Article 6(3) and profiling | Whether a derogation is available and whether profiling removes it. |
Start with intended purpose, not the model
Intended purpose is what the system is designed and documented to do, and in which workflow. Classification depends on this purpose and on how much the summaries influence decisions about individuals' access to essential services or benefits. It does not depend on the fact that the copilot uses an LLM.
Once the purpose is clear, the next question is Annex III fit: whether that purpose falls within a listed high-risk category.
How Article 6(3) and the profiling carve-out apply
A copilot that only prepares material for a human adjuster may fall within an Article 6(3) derogation.
Two questions apply: whether a derogation is available, and whether profiling removes it. A system that profiles individuals is more likely to be high-risk.
Preparatory summarization versus decision-driving summarization
The central question is whether summaries prepare a human decision or effectively determine it.
| Aspect | Preparatory summarization | Decision-driving summarization |
|---|---|---|
| Role of the copilot | Only prepares material for a human adjuster. | Effectively drives eligibility, pricing, or claim outcomes, or profiles individuals. |
| Likely classification | May fall within an Article 6(3) derogation. | More likely to be high-risk. |
What documentation supports the determination
Document the reasoning either way, whether the copilot is judged high-risk or not. The record should cover each of the classification inputs:
- The intended purpose, and the workflow in which the copilot is used.
- Whether that purpose falls within an Annex III high-risk category.
- Whether summaries prepare a human decision or effectively determine it.
- Whether an Article 6(3) derogation is available, and whether profiling removes it.
Runtime controls needed whichever way you classify
Classification decides which legal obligations apply. It does not decide whether the system needs operational control. If the copilot is high-risk, the obligations include risk management, logging, human oversight, and transparency, with duties that differ between providers and deployers. If it is not, a defensible non-high-risk position still depends on the system behaving as documented, which requires evidence from production.
Four controls apply in both cases:
- Logging of model and tool-call activity gives a record of what the copilot accessed and produced.
- Permissions, applied on least-privilege terms, limit which claims data and tools it can reach.
- Policy enforcement at runtime keeps it within its documented purpose, for instance by blocking outputs that score or characterize claimants.
- Human review checkpoints keep the preparatory role real, not nominal.
Which of these controls satisfies a given Article should be confirmed against the provider and deployer provisions.