Running an AI Incident Tabletop Exercise: A Practical Guide

What an AI Incident Tabletop Exercise Is (and Isn’t)

An AI incident tabletop exercise is a structured, discussion-based simulation that tests whether an organization’s people, processes, and runtime controls can detect, escalate, and contain an incident involving an AI agent. It differs from a traditional cybersecurity tabletop in the failure modes it scripts and the controls it tests. Traditional exercises typically simulate network intrusion, credential compromise, or ransomware, evaluating perimeter defenses, access logs, and incident response playbooks built around known IT attack patterns.

An AI agent incident often has a different origin. NIST’s AI Risk Management Framework organizes AI risk management into four functions, Govern, Map, Measure, and Manage, which provide a useful structure for assigning roles and evaluation criteria in a governance-focused tabletop rather than borrowing wholesale from generic IT templates. MITRE ATLAS catalogs adversarial tactics and techniques specific to AI and machine learning systems, including prompt injection, model manipulation, and data exfiltration through model outputs, giving exercise designers a vocabulary traditional IT threat models do not provide.

The distinction matters operationally. A traditional tabletop tests whether an access control list held. An AI incident tabletop tests whether a tool call was authorized before execution, whether an action can be attributed to the agent’s own decision versus a human instruction, and whether a natural-language input caused an unintended action without any credential ever being compromised.

AI-Specific Failure Modes to Script Into Scenarios

Scenario design should be grounded in named, documented risk categories rather than improvised threats. The OWASP Top 10 for LLM Applications lists Excessive Agency and Insecure Plugin Design as risk categories specific to agentic and tool-using systems, distinct from conventional access-control misconfiguration. Excessive Agency describes an agent granted broader permissions than its task requires; a scenario built around this category should test whether an out-of-scope action is actually blocked by a runtime policy, not assumed to be blocked because a policy document exists.

Prompt Injection is also a top-ranked OWASP category, and it differs from traditional injection attacks because the attack surface is natural-language input rather than code syntax. A realistic scenario should include an inject in which a manipulated input causes an agent to attempt an unauthorized tool call, testing whether monitoring can detect the resulting action even though no credential was compromised and no malicious code was executed.

Model Context Protocol connections introduce a further AI-specific surface. Anthropic’s MCP specification defines how AI assistants connect to external tools and data sources, and this creates a trust boundary between an agent and external systems that conventional IT incident playbooks do not model. Scenarios should include a misconfigured or overly permissive MCP connection as an in-scope asset, since each new tool integration introduces untested attack surface.

Failure modes at a glance

Feeding Results Back Into Governance

A tabletop exercise that ends with a list of technical findings and no policy update has limited governance value. NIST’s AI RMF Govern function implies that exercise outcomes should feed back into documented AI governance policy, not remain isolated observations from a single session. Joint guidance from NSA, CISA, and international partners on deploying AI systems securely frames incident response planning, including logging and monitoring, as a required component of secure AI deployment, supporting treatment of tabletop results as compliance documentation rather than an informal exercise.

Because AI-specific risk categories named in OWASP’s LLM Top 10 and NIST’s Generative AI Profile differ from traditional IT risk taxonomies, governance teams should map tabletop findings explicitly to these categories when updating policy, rather than filing results under generic IT risk headings. This mapping makes it possible to demonstrate, during an audit, that a specific documented risk, such as excessive agency, was tested and that a control gap was identified and closed.

Recurring exercises matter more for AI agents than for static IT systems, because each new tool integration or MCP connection changes the attack surface. A governance program that runs a tabletop exercise once and treats the results as permanent will miss risk introduced by later changes to agent permissions or tool connections.

Some of the runtime controls a tabletop exercise is designed to test, such as agent identity, least-privilege permissions, tool approval workflows, and audit logging, are the same controls a runtime governance platform is built to enforce continuously. Trussed AI provides runtime governance and security for enterprise AI agents, including runtime policy enforcement, MCP security, and audit logging, giving governance teams an existing control surface to validate during a tabletop exercise rather than building instrumentation solely for the exercise itself. The exercise remains valuable independent of any specific platform, since its purpose is to test whether existing controls, whatever their source, function under a simulated incident.