Implementation Guide | Healthcare

    Stopping Prompt Injection in Retrieved Clinical Documents from Triggering LangGraph Tool Calls

    Detection of injected text in clinical documents cannot be made fully reliable, so the goal is to limit what a compromised model turn can do. Treat retrieved content as untrusted data, scope each agent's tools and arguments, and authorize every model-proposed tool call outside the model. Require bound human approval for high-risk actions, and record each allow, redact, block or escalate decision as evidence.

    Control points in the graph

    1. 1

      Retriever node

      Tag every retrieved passage with its origin and mark it as data. Keep it separated from system instructions in the prompt and in state.

    2. 2

      Graph state

      Record whether untrusted content is present in the current context, so later steps can change what is permitted.

    3. 3

      Conditional edge

      Route every proposed tool call to a check step before the tool node, not directly to execution.

    4. 4

      Check step before the tool node

      Validate the tool name and arguments, and verify authorization against the caller, the operation and the target.

    5. 5

      Tool implementation

      Run each tool with the narrowest credentials it needs, so a bypassed check still has limited reach.

    6. 6

      Loop limits

      Cap chain depth, retries, tokens and cost so a hijacked loop cannot recurse indefinitely.

    Tests to run before release

    • An unauthorized tool is denied even when the model requests it confidently.
    • A tool call with an out-of-context patient identifier or recipient is rejected at argument validation.
    • A poisoned referral letter cannot trigger a send, write or external call without approval.
    • A replayed or reused approval is refused, as is one for a different call or actor.
    • Chain depth, retry and cost limits stop a looping, hijacked agent.
    • PHI does not leak through tool arguments, citations, logs or final output.

    How injected text reaches a tool node

    Generative systems combine the data channel and the instruction channel. NIST's adversarial machine learning draft notes that attackers can use the data channel to affect system operations by manipulating resources the system interacts with. Indirect prompt injection is mounted by a third party rather than the primary user, and in many cases the primary user is the one harmed.

    In clinical settings, the resources are referral letters, discharge summaries, scanned records and third-party documents. Any of these can contain text written by someone outside your trust boundary, or text mangled by OCR into something that reads like a command. OWASP lists documents, emails, database records and API responses as common carriers, and in agents that read files and call APIs, every external data source becomes a potential attack vector.

    The propagation path in a typical LangGraph agent is short. A retriever node fetches documents and writes them into graph state. A model node reads that state and emits a proposed tool call. A conditional edge sees the tool call and routes to a tool node, which executes it. Tool results are appended to the context, so the next model turn can be influenced again. If the injected text persuades the model to propose a call to read another patient's record, send a message or write to a system, nothing in that default loop asks whether the call is authorized. The model's decision is the only gate.

    A layered implementation pattern

    Start with privilege minimization, which OWASP names as the most effective control. Give each agent instance a per-context tool allowlist containing only what its task requires. A summarization agent that reads discharge summaries should not hold a tool that sends messages or writes to the record. If a tool is not bound to the agent, an injected instruction cannot invoke it.

    Next, treat model output as untrusted. The model deciding to call a tool is not the same as the user being authorized to use it. Authorization should be derived from the caller's authority, the proposed operation, the target and the arguments, and not from keyword matches in the prompt. Validate arguments against schemas and allowed values. For example, a patient identifier argument should have to match the patient in the current encounter context, and a recipient argument should have to come from an approved list.

    Finally, restrict tool invocation when untrusted data enters the context. MITRE ATLAS recommends this, and it fits graph state naturally: once retrieved third-party content is present, read-only tools stay available while transmitting or modifying tools require escalation. Content scanning can be added as a supplementary layer, but it should never be the layer you rely on.

    • 1. Mark retrieved content as data: Separate it from instructions by design, not only by prompt wording.
    • 2. Allowlist tools per context: Bind only the tools the task needs to each agent.
    • 3. Validate arguments: Check every proposed call against allowed actions and parameters.
    • 4. Authorize outside the model: Decide on caller, operation, target and arguments.
    • 5. Escalate high-risk actions: Send, write, delete and external calls need approval.
    • 6. Record each decision: Keep evidence of what was permitted and what ran.

    Designing human approval that injection cannot satisfy

    OWASP observes that an agent that asks before sending, deleting or modifying creates a checkpoint injection alone cannot bypass. That holds only if the approval is itself resistant to manipulation. A boolean such as a user_confirmed field in state is not sufficient, because the model or a poisoned document can influence state.

    The approval component should verify that the approval belongs to the current actor and to the exact tool call, that it remains valid, and that it has not already been consumed. In practice that means the approval request shows the tool, the target and the arguments as they will execute, binds the approval to a hash of that call, and expires it after one use. In a clinical workflow, also decide which actions require approval at all. Calls that transmit PHI outside the organization, write to a system of record, or are influenced by retrieved third-party content are reasonable defaults. Approving everything trains reviewers to click through, so keep the escalation set narrow and meaningful.

    Enforcing policy at runtime and producing evidence

    The checks above can be written directly into your graph, and for a single agent that is a reasonable start. As the number of agents, tools and teams grows, the same rules tend to be reimplemented inconsistently. Enforcing them in a shared execution path keeps decisions uniform and keeps them out of model-visible code.

    Trussed is one way to do this. It enforces policy in the execution path on individual interactions and tool calls, with outcomes of allow, redact, block, rate limit or escalate for human approval. Enforcement happens at runtime, not in a review after the fact. For PHI protection, redaction can remove sensitive values from an interaction before it reaches a tool or leaves the boundary, while a block stops a call that falls outside policy. Each decision produces audit-ready evidence of what the agent was allowed to do and what it did, mapped to HIPAA including 164.312(b) audit controls. Trussed can be self-managed in the customer cloud, with configurable retention and data residency, and it runs in production with healthcare and pharma organizations.

    The scope is interactions and tool calls. It does not replace sound tool design, least-privilege credentials or schema validation in your own code, and it is not a substitute for testing.

    Where deterministic control sits in a tool-calling loop

    Retrieval

    Clinical documents enter as untrusted data, never as instructions.

    Model turn

    May be steered by embedded text. Its output is treated as untrusted.

    Tool-call boundary

    Allowlist, argument validation and authorization run outside the model.

    Approval and policy

    High-risk calls escalate. Every decision is recorded.

    Enforce tool-call policy where the model cannot override it

    See how policy enforcement in the execution path and audit-ready evidence apply to LangGraph tool calls in healthcare.

    Explore Agent and Tool-Call Enforcement