Preventing OpenAI Web Search Tool Calls from Leaking Customer Context into Search Queries
Treat every model-generated search query as an untrusted outbound payload. Reduce what the model can see, limit when it can search, and constrain prompts, but do not rely on these alone because they are non-deterministic. Add a runtime check at the tool-call boundary that detects sensitive content in the query, then allows, redacts, blocks or escalates it, and records the decision as evidence.
Enforce policy at the tool-call boundary
- 1
Allow
Query matches policy and is sent unchanged.
- 2
Redact
Sensitive spans are removed or masked and the modified query proceeds.
- 3
Block
The call is stopped and the agent receives a controlled response.
- 4
Escalate
A reviewer sees the full query and approves or rejects that exact call.
Designing the layered approach
- Minimize context: scope retrieval by access control and clear memory between data subjects.
- Scope tools: expose search only to agents that need it and deny unknown tools by default.
- Constrain prompts, and treat them as a weak layer rather than a control.
- Inspect the actual query string at the tool call, not only prompts or responses.
- Set the action per tool and per policy, and fail closed when detection fails.
- Test for false positives and false negatives, and review them on a regular schedule.
How customer context reaches a search query
When a model uses a web search tool, it composes the outbound query itself. Your application does not write that string. The model does, and it can draw on anything in its context window.
In multi-turn and agentic systems that context accumulates. Prior conversation turns, retrieved documents, tool outputs, API responses and customer records all sit in the same window. A model that decides it needs outside information may fold an account number, a name, a diagnosis or an internal project name into the query because it looks like a useful search term. Nothing in that behavior is malicious. It is the model optimizing for a relevant query using everything available.
The result is that the search service receives text your team never reviewed. Security teams should treat the query the way they treat any other outbound payload that crosses the enterprise boundary: as data that needs inspection before it leaves.
Leakage patterns to expect
Three patterns cover most cases. The first is incidental inclusion, where the model copies identifying details from the conversation or a retrieved record into a query because they seem relevant. The second is context bleed in long sessions, where content from an earlier task or a different data subject surfaces in a later query. The third is deliberate exfiltration through indirect prompt injection.
Indirect injection embeds instructions in content the model retrieves, such as webpages, documents, emails, database records or API responses. An agent that browses the web makes every external source a potential attack vector. A planted instruction can tell the model to encode sensitive data into what looks like an ordinary tool call. OWASP names search queries and email subjects as typical carriers for this kind of exfiltration through legitimate channels.
Because instructions and data share a single input stream, OWASP treats this as an architectural condition rather than a bug that can be patched. That shapes where controls should sit.
- Incidental inclusion of identifiers from conversation or retrieved records
- Context bleed across turns, tasks or data subjects
- Indirect injection instructing the model to place data in a query
What application-layer mitigations can and cannot guarantee
Application-layer controls are worth doing, and each reduces exposure. Context minimization means the model sees less to leak: enforce data access controls at retrieval and integration layers, apply data classification, keep credentials and sensitive data out of system prompts, and purge memory when the data subject changes. Tool configuration means granting the agent only the tools it needs, treating unknown tools as denied, and constraining where and when search can be invoked. Prompt design means instructing the model not to include personal or confidential details in queries.
The limits matter. Minimization only helps for data you know is in the window and can remove without breaking the task. Tool scoping decides whether search is available, not what a given query contains. Prompt constraints are probabilistic. OWASP notes that system-prompt restrictions may not always be honored and can be circumvented through prompt injection or other vectors, and recommends enforcing controls outside the model instead of relying on confidentiality instructions.
The practical conclusion is that these measures lower the likelihood of leakage but cannot provide a deterministic guarantee for a string the model writes at runtime.
Evidence of what was permitted, modified or stopped
Enforcement is only useful to an auditor if each decision can be shown afterward. For every search call, the record should capture the policy applied, the actor, the outcome, and the relationship between the original and the modified query. Approval records should tie to the specific actor and tool call.
There is a design tension. Evidence that stores the original query can become a new store of sensitive data. Record security-relevant metadata and decisions, keep unnecessary sensitive content out, and set retention and access controls on what remains.
Trussed enforces policy in the execution path on individual tool calls, including allow, redact, block, rate limit and escalation for human approval, and generates audit-ready evidence from that enforcement. Evidence maps to frameworks including HIPAA, NAIC, NYDFS, EU AI Act and NIST. It supports self-managed deployment in the customer cloud, with configurable retention and data residency, which bears directly on where query evidence is stored.
Where a search query can be controlled
Context window
Conversation history, retrieved documents, tool outputs and customer records share one input.
Model-composed query
The model writes the search string itself, drawing on anything it can see.
Policy check on the tool call
The query is inspected before it leaves the boundary: allow, redact, block or escalate.
Decision evidence
Each outcome is recorded with the policy applied and the actor.
Put a policy check on every outbound search query
See how policy enforcement on tool calls works for a regulated enterprise, and what audit-ready evidence each decision produces.
See How Evidence Is ProducedHow does your AI governance program compare?
See where your program has gaps in less than 2 minutes.