Detecting EU AI Act serious incident candidates from runtime telemetry

What the definition asks and what telemetry can see

Article 3(49) of Regulation (EU) 2024/1689 defines a serious incident as an incident or malfunctioning of an AI system that directly or indirectly leads to specified outcomes. The first two are the death of a person or serious harm to a person's health, and a serious and irreversible disruption of the management or operation of critical infrastructure. The recital text also describes infringements of Union law obligations that protect fundamental rights, and serious damage to property or the environment.

The definition is outcome-based. Runtime telemetry sees the first half: an incident or malfunctioning of the AI system. It often cannot see the second half, the harm itself, because the outcome happens in a clinic, a plant, a customer's account or a courtroom, not in your logs. The words "directly or indirectly" widen the gap further, since the link between a system event and a downstream harm may not appear in the system's own records.

This shapes the whole design. Treat telemetry as a source of candidate signals and as input to a human determination. As of November 2026, the provider's reporting trigger is a causal link between the AI system and the serious incident, or the reasonable likelihood of one. That is a judgment, and it belongs to people with the authority to make it.

Map definition outcomes to observable signals

Start with one row per outcome in the definition and write down which runtime events could indicate it, and which could not. The signals below are design choices for your environment, not regulatory requirements. Mark outcomes that are rarely observable from telemetry alone, so nobody assumes silence in the logs means no incident.

• Health harm: Outputs or tool calls in a clinical or safety context that were wrong, unsafe or acted on despite a block or escalation. The harm itself usually arrives through an external report.
• Critical infrastructure disruption: Failed or repeated tool calls against operational systems, unexpected state-changing actions, or rollbacks. Whether the disruption was serious and irreversible is rarely visible in telemetry.
• Fundamental-rights infringement: Policy decisions that show sensitive data exposure, discriminatory output patterns flagged by review, or consequential decisions made without required human approval.
• Property or environmental damage: Actions against physical or financial systems that executed outside expected bounds. Damage is normally established from outside the AI system.
• Malfunctioning indicators: Error spikes, output anomalies, policy bypass attempts that succeeded, and human overrides that reverse an AI action.
• External reports: Deployer, user or third-party complaints entered as candidates with the same record format as telemetry-derived ones.

Define rules and severity thresholds that limit noise

Most blocked prompts, timeouts and policy denials are routine. If every one becomes a candidate, reviewers stop reading. Separate routine events from candidate events by combining signals, not by alerting on any single one.

A workable pattern uses three tiers. Tier one is routine: a policy block that worked as intended, with no downstream action. Tier two is an anomaly: a malfunction in a high-risk use case, a repeated failure, or a human override of an AI action. Tier three is a candidate: an anomaly in a high-risk system plus an indication of possible harm, such as an executed action with a real-world effect, an external complaint, or a sensitive-data exposure. Correlate by system, user context, session and time window, and raise severity when signals cluster.

Tune thresholds against your own history and review them regularly. Err toward over-escalating borderline events to a reviewer rather than discarding them, because the cost of an unnecessary review is lower than the cost of a missed candidate. Keep a record of the thresholds in force at the time of each decision.

Where runtime enforcement fits, and what stays outside

Policy enforcement in the execution path produces decision records as a by-product of normal operation. When each interaction or tool call is allowed, redacted, blocked or escalated for human approval, the decision, the actor and the approval outcome are recorded at the moment they happen, rather than reconstructed after the fact. Trussed enforces policy at that level across models, agents, tools and MCP servers, and produces audit-ready evidence mapped to the EU AI Act. It can be deployed self-managed in the customer cloud, with configurable retention and data residency, which supports the storage decisions above.

The scope is individual interactions and tool calls. That gives you strong candidate signals and decision records, but it does not give you business transaction-level governance, end-to-end tracing of agentic workflows, or model provenance. Where a candidate depends on those, other records have to fill the gap.

Several things remain outside runtime telemetry whatever tooling you use: harms that surface in the physical world or in downstream systems, the causal link between a system event and an outcome, the decision on reportability, and the definition of awareness. Build those into the process explicitly. A detection capability that claims to answer them from logs alone will give false assurance.