How to Prevent an AI Customer Service Agent from Disclosing Account Details Before Identity Verification
Make disclosure depend on a verification state held by the system, not on prompt instructions. Apply a policy decision at three points: before each tool call, before retrieved data enters model context, and on model output. Each decision allows, redacts, blocks or escalates, and is recorded as evidence.
Evidence the control should record
- Interaction and tool-call identifiers, with timestamps
- Verification state at the moment of the decision, and how it was set
- Policy name and version that was applied
- Tool, parameters and the account the call targeted
- Decision taken: allow, redact, block or escalate
- Data fields affected, and the approver and outcome for escalations
Where account data leaves an agent before verification
An LLM-based customer service agent is usually connected to CRM, billing and account lookup tools. Because the model decides which tool to invoke, when, and with what parameters, a caller who has not been verified can still cause a lookup to run. OWASP's MCP guidance describes this as a new attack surface, since agents execute tools from natural language against sensitive systems.
There are four disclosure paths worth mapping. First, tool calls that return account data, where the model chooses the tool and its parameters. Second, retrieval or lookup results placed into model context. Third, model output that reproduces data from those results. Fourth, error messages and configuration details, which OWASP also identifies as leakage paths.
The bypass routes are familiar. Prompt injection can steer the model into calling a tool it should not. Social engineering works on agents as it does on people: call centers are a known source of account takeover because attackers talk their way past verification checks. NIST material adds that an agent may have access to data sources or tools that the user it serves would not normally reach, so the agent's own credentials become part of the exposure.
Why prompt instructions and after-the-fact review are not enough
An instruction such as "never share account details until the caller is verified" is a request to the model, not a control. It is processed as text alongside the caller's input, so injected or persuasive text can compete with it. OWASP's prompt injection guidance points instead to controls that hold regardless of what the injected text says: enforced tool permissions, parameter validation and least privilege.
A second weakness is where the filter sits. Data that enters model context can reach the caller even if the final response is later filtered, because the model may paraphrase or summarize it. Applying the decision before data enters context, as well as on output, follows from the principle in NIST's agent identity guidance that each tool invocation should be verified at runtime against current state.
Post-hoc review has a different limit. Reviewing transcripts can show that a disclosure happened, but it cannot undo it. Review is useful for tuning policy and for investigations, but it should not be the mechanism that decides whether a caller sees a balance or an address.
Define a verification-gated disclosure policy
Start with the policy, because enforcement can only be as precise as the rules it applies. The sources do not prescribe particular states or fields, so these are design decisions for your team.
Treat verification state as a value held by the system and set only by the verification step. Nothing the caller or the model says in conversation should be able to change it. Then map each state to permitted tools and data fields, and match an action to each pre-verification scenario.
- 1. Define states: For example unverified, partially verified and verified. Keep the set small and make transitions explicit.
- 2. Map tools and fields to states: List which tools may run and which fields may be returned in each state. Default to deny for anything unlisted.
- 3. Block pre-verification lookups: An account-lookup call before verification, or for an account the caller has not verified against, is a policy violation and is blocked.
- 4. Redact partial results: If a tool returns more fields than the current state permits, remove or mask the excess fields rather than failing the whole interaction.
- 5. Escalate high-risk requests: Route requests such as changing contact details to human approval before they run.
- 6. Allow non-sensitive responses: General product or process answers need no gate and should not be slowed by one.
Apply the decision at each point in the execution path
Place enforcement outside the model's text reasoning, so the decision does not depend on how the model interprets a message.
At the tool call, evaluate every proposed invocation with the current verification state, the tool, its parameters and its risk. This is a per-call decision, not a session-start check. Validate parameters, and screen each call against the caller's original intent, as OWASP recommends for agents with tool access. Give the agent least-privilege credentials, preferably read-only, with separate scopes for pre-verification and post-verification work. If the agent delegates to sub-agents or other tools, the downstream scope should not exceed the parent's.
At retrieval, filter fields before results reach model context. This is the point that most often gets skipped, and it is the one that stops the model from ever seeing data it should not repeat.
At output, inspect the response for account data and redact or block what the state does not allow. OWASP notes that pattern matching can detect and redact confidential content, though its guidance is framed largely around training data. Pattern-based redaction on output is a useful backstop, but it should not be your only control, because pattern matching will miss data that is paraphrased or formatted unexpectedly.
Finally, test before release with prompt injection and social engineering attempts made while the caller is unverified. Include requests that try to assert verification in conversation, and confirm that they do not change state.
Enforcing this at runtime with Trussed
Trussed is one way to apply these decisions. It enforces policy in the execution path across models, agents, tools and MCP servers, with the actions allow, redact, block, rate limit and escalate for human approval. Its scope is individual interactions and tool calls, which matches the per-call decisions described above. The enforcement itself produces audit-ready evidence, and it can be deployed self-managed in the customer's cloud with configurable retention and data residency.
Trussed does not perform the identity verification itself, and it does not replace least-privilege credentials or testing. Whatever you choose, evaluate it on a few questions: does enforcement occur at the tool-call boundary and on output, where does verification state come from and can the model or caller alter it, and what is recorded for each decision.
Verification-gated disclosure path
Tool call
Account lookups are checked against verification state before they run.
Retrieved data
Fields are filtered before they enter model context.
Model output
Responses are screened for account data the caller is not cleared to see.
Evidence
Each decision is recorded with policy version, state and affected data.
See how enforcement and evidence work for agent tool calls
Walk through a verification-gated policy and the evidence it produces with a Trussed expert.
See How Evidence Is ProducedHow does your AI governance program compare?
See where your program has gaps in less than 2 minutes.