AI Security Incidents: Prompt Injection, Jailbreaks, and Data Leakage

Why these incidents matter in agent architectures

Security engineers evaluating production AI agents face a different control problem than static chat. Agents follow a plan, select tools, fill arguments, observe results, and continue. The model is not only generating text; it is mediating which APIs run and with what parameters. When untrusted input reshapes that loop, the impact is operational: policy bypass, privilege escalation through tools, and loss of sensitive context.

Industry guidance consistently ranks prompt injection among the leading risks for LLM applications, covering both direct overrides in the user channel and indirect injection via external content. Jailbreaks weaken refusal and safety behavior so attacker-controlled instructions can proceed. Sensitive-information disclosure is treated as a primary risk in its own right, often enabled by injection but also possible through careless context assembly. Providers and standards bodies note that system prompts and model safety filters alone are insufficient against determined attacks. Layered controls outside the model are required.

Direct versus indirect prompt injection

Direct prompt injection places attacker-controlled text in the user or equivalent message channel with the intent to override system or developer instructions. The technical pattern is straightforward: the model cannot reliably distinguish peer natural-language content from privileged policy text once both sit in the same context. Instructions such as ignore prior rules, change the objective, or invoke a different tool compete with enterprise policy on equal footing inside the model.

Indirect prompt injection is the pattern that matters most for retrieval-augmented and tool-using enterprise agents. Malicious instructions are embedded in untrusted artifacts that the system itself pulls into context: RAG documents, web pages, email or ticket bodies, uploaded files, or tool-returned JSON. The user may have issued a benign request. The model nevertheless treats retrieved or observed text as actionable instruction, altering goals, tool selection, or output without explicit user intent.

What distinguishes the two for defenders is the trust boundary and the entry path. Direct injection is largely an input-channel and session problem. Indirect injection is a content supply-chain and observation problem: every retrieval corpus, connector, and tool response is a potential instruction surface. Architectures that concatenate untrusted content as peer instructions without labeling, segmentation, or filtering expand that surface by design.

Jailbreaks and unauthorized tool use in multi-step workflows

Jailbreak techniques craft single-turn or multi-turn inputs that defeat model or system safety alignments so the model ignores developer or enterprise policies. In a pure chat setting, the result may be disallowed content. In an agent setting, the result can be unauthorized tool invocation, argument tampering, or policy bypass across a multi-step plan.

Because the agent loop is plan, select tool, fill arguments, observe, continue, a successful jailbreak does not need to break every control at once. Compromising the planner is enough if tool execution trusts the model’s choice of action and parameters. The agent may chain tools, use intermediate observations as further instruction, and amplify impact through autonomous tool sequences and persistent memory. Agentic systems increase the blast radius of injection precisely because compromised reasoning loops can keep acting until an external gate stops them.

Data leakage paths in production agents

Leakage paths in production deployments are structural, not accidental. Unrestricted context windows can regurgitate secrets, PII, or regulated data that were retrieved, remembered, or passed as tool results. Conversation memory and cross-session retrieval can surface material outside the original access boundary. Tool responses may be echoed, summarized, or forwarded to secondary tools such as email, HTTP, or code execution, moving data out of the intended trust domain. Verbose tool errors can expose configuration or credentials. Logs that capture full prompts and outputs without classification controls create a secondary leakage channel during investigation or support.

OWASP and provider guidance treat sensitive-information disclosure, including PII and secrets in training data, context, or logs, as a primary LLM risk. It is distinct from injection but frequently enabled by it: an injected instruction to summarize the full context, call an outbound tool with retrieved chunks, or ignore redaction rules converts a containment failure into exfiltration.

Constraining context is therefore a first-order security control. Redact secrets before they enter the window. Bound retrieval. Segment memory by tenant and user. Strip or annotate instruction-like content in tool observations. Apply DLP-style scanning on inbound context and outbound model and tool responses. Avoid placing long-lived secrets in prompts or agent memory; bind tools to scoped, short-lived credentials tied to a principal.

Trust boundaries and runtime control placement

Effective designs separate trust boundaries for system instructions, developer policies, user input, retrieved context, and tool outputs. Untrusted content should never be merged as an undifferentiated peer of policy text. Retrieved and tool-returned material should be labeled, filtered, and isolated before it re-enters the model. The policy engine that authorizes tools must sit in front of execution, not inside the prompt.

Runtime enforcement outside the model is what reduces blast radius when injection or jailbreak succeeds. Authenticated caller identity for each agent and each invocation, scoped API credentials per tool, allow and deny policies on tool names and argument shapes, rate limits, data-classification checks, and immutable audit logs of prompts, retrieval identifiers, tool invocations, policy decisions, and outputs form the minimum practical set. High-risk tools such as code execution, payments, outbound network, and admin APIs should use allowlists and step-up or dual-control approval. Providers document the same layering: input and output filtering, tool sandboxing, and human approval for high-risk actions, in addition to model-side safety.

Implementation checklist for containing incidents

Operational programs should encode the above as deployable controls and testable gates, not prompt-only guidance.

• Map injection, jailbreak, and leakage scenarios into existing risk frameworks and incident response runbooks so agent failures are handled with the same rigor as other security events.
• Establish ownership for agent identities, tool inventories, and prompt or policy change control with auditable approvals.
• Retain investigation-grade logs while applying retention and access controls so the log store does not become another disclosure path.
• Align context, memory, and tool data handling with organizational classification and privacy rules, including strict limits on cross-tenant context sharing.

Evaluation criteria for runtime governance

• External tool-call enforcement: Confirm allow and deny rules and argument validation execute outside the model so a jailbreak cannot self-approve actions.
• Identity-bound invocations: Require a least-privilege principal and short-lived credentials on every agent and tool call.
• Untrusted content isolation: Verify RAG and tool outputs are labeled, filtered, and constrained before re-entering the context window.
• Audit completeness: Demand runtime telemetry for prompts, retrieval IDs, tool calls, and policy denials, with clear access control on the trail itself.
• Pre-production red teaming: Cover direct and indirect injection and multi-step exfiltration paths with automated or manual suites before production.

Governance and residual risk

NIST adversarial machine learning guidance frames prompt-based attacks among threats that require ongoing evaluation and mitigation, not one-time hardening. Public sources describe attack classes and defenses clearly but do not provide a single authoritative enterprise frequency statistic. Effectiveness of prompt-level defenses also varies by model and release. That uncertainty is itself a design input: assume model-side refusal will fail under adversarial pressure, and design so failure does not equal unconstrained tool access or unbounded context disclosure.

Vendor claims about guardrails differ in architecture. Independent evaluation against injection and multi-step agent exfiltration paths is required. For security engineering teams, the practical standard is simple. Can an untrusted planner still move data or invoke privileged tools when the prompt is hostile? If the answer depends only on the model complying, the control plane is incomplete. Runtime identity, permissions, tool governance, monitoring, and auditability are what make agent deployments operable under real incident conditions.

Trussed AI focuses on runtime governance and security for enterprise AI agents, including agent identity, least-privilege permissions, tool approval workflows, and audit logging. Whatever control plane you operate, the engineering goal remains the same: keep policy enforcement and evidence collection outside the model so injection, jailbreak, and leakage incidents stay contained and investigable.