How to Build an AI Incident Severity Matrix: Levels and Triggers
An AI incident severity matrix classifies AI agent runtime events, such as unauthorized tool calls, permission escalations, or policy violations, into defined severity levels based on observable, machine-readable triggers rather than subjective judgment. Each level maps to a specific escalation path, from automated containment to human review, and produces the audit record needed for governance and regulatory reporting.
Structuring the Matrix: Trigger Type vs. Impact Category
A defensible severity matrix separates the runtime signal that fired from the consequence it represents, then ties each combination to a specific, auditable escalation action.
- 1
Separate trigger from impact
Classify by the runtime signal (tool-call anomaly, permission escalation, policy violation) independently from the impact it causes (data, safety, compliance, operational).
- 2
Map escalation to enforcement points
Tie severity determination and containment action to the same runtime component, such as the policy engine, gateway, or orchestrator, so the event record is auditable end to end.
- 3
Version severity definitions
Tie each severity definition to the runtime policy version in effect at the time of the incident to support later audit reconstruction.
Core Building Blocks of a Defensible Severity Matrix
Four elements recur across every well-formed severity matrix, regardless of the specific triggers an organization defines.
Trigger Type
The specific runtime signal that fired, such as a tool-call anomaly or permission escalation.
Impact Category
The consequence the trigger represents: data, safety, compliance, or operational impact.
Escalation Path
The enforcement point and action tied to each severity level, from logging to containment.
Audit Record
The documented signal, policy version, and decision rationale supporting later review.
Governance and Audit Requirements
A severity matrix is only as defensible as the evidence trail it produces. These requirements support both internal governance and external audit or regulatory review.
- Regulatory alignment: the EU AI Act's post-market monitoring and serious incident reporting obligations for high-risk systems require documented processes; the matrix should generate the evidence needed to support that reporting.
- Management system alignment: ISO/IEC 42001 expects a formal AI management system with incident handling procedures; a severity matrix can operationalize that requirement.
- Audit defensibility: document why a severity level was assigned, including the specific runtime signal and the policy version in effect, not just the resulting classification.
- Assigned accountability: explicitly name which roles hold escalation and containment approval authority to avoid diffusion of responsibility during live incidents.
What an AI Incident Severity Matrix Is (and Isn't)
An AI incident severity matrix is a structured framework that classifies AI agent runtime events, such as unauthorized tool calls, permission escalations, policy violations, or unexpected data exposure, into defined severity levels tied to specific escalation actions. It differs from a general risk register or policy document because it operates on discrete, observable triggers rather than qualitative risk descriptions.
No major standards body currently publishes a ready-made severity taxonomy for AI agent runtime behavior. NIST's AI Risk Management Framework (AI RMF 1.0) establishes a voluntary structure for identifying and managing AI risk across the lifecycle, but it does not define incident severity levels or thresholds. Traditional cybersecurity incident models, such as NIST SP 800-61 Rev. 2, classify severity using functional impact, information impact, and recoverability effort. These factors remain useful conceptually but were developed before autonomous agents existed and do not account for signals unique to agent runtimes, such as tool-call behavior or permission scope. Enterprises building a severity matrix for AI agents must adapt these general models rather than adopt any single one directly.
Why Traditional IT Severity Models Fall Short for AI Agents
Traditional IT incident severity models were built around network and system-level signals such as outages, breaches, and data loss. AI agent runtimes introduce signal types that do not map cleanly onto that framework.
Tool-call anomalies, including unexpected function invocation, parameter tampering, or abnormal call frequency, are a distinct signal class absent from conventional IT monitoring. Permission or scope escalation, where an agent invokes capabilities beyond its granted role or credentials, parallels privilege escalation in cybersecurity but requires logging of intent versus action to be meaningful. Policy violations also differ depending on whether guardrails are enforced pre-execution, blocking the action, or post-execution, logging it after the fact; this distinction determines whether an event is a near miss or a realized incident.
Data exposure severity depends on the exposure channel: output generation, tool result leakage, or memory and context persistence each require different detection instrumentation and carry different downstream risk. Model misbehavior, such as hallucination, goal drift, or unsafe recommendations, is harder to bound with binary triggers and typically requires confidence or deviation thresholds rather than a single yes/no condition. Risk categories such as excessive agency, insecure output handling, and sensitive information disclosure, catalogued in the OWASP Top 10 for LLM Applications, along with adversarial techniques catalogued in MITRE ATLAS, provide useful reference categories for defining these triggers, though neither source prescribes severity levels directly.
Automated Containment vs. Human Review
Not every triggering condition warrants the same response. Objectively measurable triggers, such as an unauthorized tool call or confirmed credential misuse, can be mapped to automated containment because the event itself is unambiguous and the containment action, blocking, revoking, or isolating, is deterministic. Ambiguous signals, such as suspected model drift or a borderline policy interpretation, are better routed to human review, since automated action on an uncertain signal risks either false containment or a missed escalation.
Decide escalation authority in advance
Escalation authority should be explicitly assigned: which role can approve automated containment, and which situations require human sign-off before action is taken. Leaving this ambiguous during an active incident diffuses accountability and slows response, undermining the purpose of having a severity matrix in the first place.
This distinction should be decided in advance, not improvised during an incident. Building that decision into the matrix itself, rather than relying on judgment calls made under pressure, is what makes the framework operational rather than aspirational.
Build a Defensible AI Incident Response Process
Trussed AI provides runtime governance for AI agents, including policy enforcement, permission controls, tool approval workflows, and audit logging that can support the runtime signals a severity matrix depends on.
Explore Runtime Governance