How to Prevent an LLM From Leaking API Keys and Credentials

How secrets and internal URLs reach model output

Leakage rarely comes from one place. OWASP lists security credentials among the sensitive data types covered by LLM02:2025 Sensitive Information Disclosure, and notes that exposure can affect both the model and the application context around it. Several pathways matter in practice.

Retrieved context is the first. OWASP's RAG guidance says a model can leak sensitive data from retrieved chunks even when everything upstream is secure. Coding assistants and agents add another: their context can hold credentials, proprietary logic, and internal architecture details, and agents may pass credentials, file contents, or environment variables as tool arguments without any awareness of data classification.

Tool results are the third. OWASP's MCP guidance says to treat every tool response as untrusted, including responses from approved servers. Shared or persistent context windows can expose one user's or agent's data to another, and secrets held in model memory, protocol logs, or debug traces can be retrieved through prompt injection. Training data is a further pathway, though OWASP's guidance there is general: sanitize data so user data does not enter the model. Internal URLs follow the same routes as keys, since hostnames and endpoints appear in the same code, configuration, and documents.

Because the pathways are mixed, no single upstream control closes them all. That is why the design needs an inspection point on the output side.

Where the control sits determines what it can see

Three placements are common. OWASP says system-prompt restrictions may give some mitigation, but they may not always be honored and can be circumvented through prompt injection or other vectors. They are a request to the model, not an enforcement point.

Application-layer filters are enforceable, but they only cover the code paths developers remember to wrap. Agents, coding assistants, and tool calls often run outside that wrapper. A control in the execution path sees each model response and each tool call and result as it passes, so the same policy applies regardless of which application issued the request. That architectural reasoning, together with OWASP's description of output validation as the last line of defense, is why this placement deserves primary weight. It should still sit alongside upstream hygiene, not replace it.

• Prompt instructions: Cheap to add and partially useful. Not enforceable, and can be bypassed. Never the sole control.
• Application-layer filters: Enforceable where deployed. Coverage depends on every application and agent path being wrapped, and tool calls are easy to miss.
• Inline in the execution path: Applies one policy to responses and tool calls across models, agents, and assistants. Requires a component that sits in the path.

Detection techniques and their limits

Use several detectors, because each fails differently.

Pattern matching with provider-specific key formats is precise for known credential types and cheap to run. It misses custom or rotated formats and anything the model has reformatted, such as a key split across lines or encoded.

Entropy analysis catches high-randomness strings that no pattern anticipates. It also flags hashes, identifiers, and generated IDs, so it raises false positives unless combined with other signals.

Context-based detection looks at surrounding text: an assignment to a variable named like a secret, an authorization header, or a connection string. Contextual classifiers extend this but are probabilistic and need tuning against your own traffic.

Internal URL detection works by matching against your own domains, private address ranges, and known internal hostnames. Its accuracy depends on how current that inventory is.

Measure false-positive and false-negative rates on your own representative traffic before setting actions. Published figures will not match your data.

False positives, human approval, and evidence

False positives will occur, and a block that cannot be reviewed teaches teams to route around the control. Send mid-confidence detections to a human approver instead of blocking outright. Bind each approval to the specific action and its arguments so it cannot be reused for a different call, as OWASP's agent guidance points toward. Feed approver outcomes back into rule tuning.

Evidence answers a different question: what was detected and what happened. For each enforcement action, retain the timestamp, the requesting user or agent, the model and tool involved, the rule or detector that fired, the confidence, the action taken, and any approver decision. Store a fingerprint or masked form of the matched value rather than the value. OWASP advises redacting secrets and PII from logs, and that includes enforcement records. Otherwise the evidence store becomes a new secret store.

This record is also what lets you show an auditor how a policy was applied to individual interactions, rather than reconstructing events from raw logs after the data has left.