How to Run an AI Agent Failure Mode Review After an Outage

Why Agent Reviews Differ From Software Postmortems

A conventional software postmortem examines code paths, deployment changes, and infrastructure state. An AI agent failure mode review has to cover the same ground plus a layer that traditional postmortems do not address: the agent's autonomy boundary at the moment it acted. NIST's AI Risk Management Framework identifies traceability, documentation of data, processes, and decisions across the AI lifecycle, as a required characteristic of trustworthy AI systems specifically because this layer is otherwise invisible after the fact.

This matters because an agent outage is rarely explained by a single defective function call. It is usually explained by a combination of what the agent was permitted to do, what context it was given, and whether a policy check existed to stop it before the action executed. Reviewing only the technical failure and skipping the permission and policy layer produces an incomplete postmortem that will not prevent recurrence.

Data Sources Required for Reconstruction

Before a review can begin, four categories of evidence need to be available. Tool-call and API invocation logs establish what actions the agent actually took. Permission and identity records establish what the agent was authorized to do at that moment, which is not always the same as what it was capable of doing. Model input and output pairs, where logging exists, establish the reasoning context that led to the action. Policy enforcement logs establish whether a runtime check ran, and if so, what it decided.

CISA's guidance on AI system security treats audit logs of AI decisions, inputs, and outputs as a prerequisite for incident reconstruction, not an optional add-on. In practice, many teams discover during their first agent-related outage that one or more of these logs either does not exist or was not retained at sufficient granularity. That gap should itself be documented as a finding, since it constrains what the review can conclusively determine.

Common Failure Categories in Agent Systems

Agent failures tend to cluster into a smaller set of categories than general application bugs, largely because the failure modes are tied to autonomy and permission scope rather than arbitrary code defects. OWASP's Top 10 for LLM Applications names Excessive Agency as a distinct category, where an agent is granted more functionality, permission, or autonomy than the task requires, leading to actions outside intended scope. OWASP's agentic AI guidance separately identifies unbounded tool invocation, where an agent calls tools repeatedly or in unanticipated sequences without a check, and the absence of human-in-the-loop checkpoints for high-impact actions.

MITRE ATLAS documents adversarial techniques that exploit these same weaknesses, including excessive permissions and tool or API misuse, which gives reviewers a second reference point for categorizing behavior that was not necessarily adversarial but exposed the same underlying weakness. Distinguishing these categories matters because the remediation differs: an excessive agency finding points to permission scoping, while an unbounded tool invocation finding points to rate limiting or approval gates.

Distinguishing Governance Gaps From Technical Gaps

One of the more consequential steps in the review is determining whether a policy existed and was not enforced, or whether no policy existed at all. These require different remediation paths. A policy that existed but was not enforced at runtime indicates an architectural gap, likely that permission checks are validated at credential issuance rather than at the moment of each action. A policy that never existed indicates a governance gap, meaning the scenario was not anticipated in the agent's permitted actions or context boundaries.

NIST's Generative AI Profile recommends organizations explicitly define and monitor permitted actions and context boundaries for generative AI systems. Reviewing against this recommendation gives teams a concrete standard to check the agent's behavior against, rather than relying on an after-the-fact judgment about what the agent should have been allowed to do. This distinction should be documented explicitly in the review output, since conflating the two leads to remediation that addresses the wrong layer of the system.

Where Runtime Controls Fit In

The findings from a failure mode review are only useful if they can be converted into enforceable controls. This is the point where governance intent has to become a runtime mechanism. If the review finds that an agent's permissions were broader than the task required, the fix is not a policy document, it is a runtime enforcement point that checks scope at the moment of each tool call. If the review finds that a high-impact action had no approval gate, the fix is a tool approval workflow that requires human sign-off before execution.

Trussed AI provides runtime governance for enterprise AI agents, including permission enforcement, tool approval workflows, and audit logging designed to support exactly this kind of reconstruction and remediation cycle. The review methodology described here does not depend on any specific platform, but having runtime enforcement and complete audit logging in place before an incident occurs determines how much of the review can be conclusively answered rather than inferred.