Routing LangChain Model Calls Through a Policy Enforcement Proxy Without Rewriting Agent Code

The proxy as an enforcement point in the model-call path

When many teams build LangChain agents, model calls usually go straight from each agent to a provider. Policy on allowed models, data handling and tool use then depends on every team implementing it the same way, and evidence of what each agent was permitted to do ends up scattered across codebases. Adding controls inside each agent means rewriting or re-instrumenting all of them, which is slow and hard to sustain.

A proxy changes where enforcement happens. NIST describes a proxy agent as an intermediary between two hosts that never allows a direct connection between them, and notes that an application-proxy gateway can offer higher security for some applications because it inspects traffic content to identify policy violations. Packet filters, by contrast, are not concerned with the content of packets. That distinction matters here. Reachability controls can decide which hosts an agent may contact, but only an application-layer intermediary can decide what a given request may contain.

NIST also separates the decision function from the enforcement function. The policy enforcement point applies a decision, and the policy decision point can sit in the proxy itself or in an external service. For a model-call proxy, decide early which pattern you want, and what the proxy does when the decision service is unreachable. That failure behavior is a policy choice, not an implementation detail.

Routing mechanisms, ordered by invasiveness

The goal is to redirect traffic with the smallest change that still gives you a reliable enforcement point. Work through the options in this order and stop at the least invasive one that covers a given workload. For each LangChain provider integration in your estate, record which of these it can use, because support differs by integration and by deployment pattern.

• 1. Endpoint or base URL setting: Where an integration exposes a configurable endpoint, point it at an OpenAI-compatible proxy address through deployment configuration rather than code. This is the most direct override and keeps the proxy visible to the team that owns the agent.
• 2. Environment variables: Where the underlying provider client reads an endpoint from the environment, set it in the container image, deployment manifest or runtime template. One platform-owned change can cover many agents.
• 3. HTTP client and proxy settings: Where the runtime honors standard HTTP proxy settings, set them at the process or container level so outbound requests flow through an egress proxy. Check how TLS trust and certificate handling work before you rely on this.
• 4. Network-level egress routing: Route or transparently redirect outbound traffic to provider addresses at the network layer, for example through an egress gateway or sidecar. This covers agents whose configuration you cannot change, but it is the hardest to operate and debug.

Compatibility issues to test before rollout

A proxy is only acceptable if agents behave the same through it. Test these areas in staging with representative agents rather than assuming they work.

Streaming responses are the first concern. A proxy that buffers a full response before releasing it will change latency and can break agents that consume partial output. Content-level redaction or blocking on a stream also needs a defined behavior: what happens when a violation is detected after part of the response has been delivered.

Tool and function calling payloads come next. Requests and responses carry structured arguments, and the proxy must pass them through intact while still being able to apply policy to them. Confirm that the proxy preserves the payload structure the agent expects.

Authentication needs an explicit decision. Either the proxy forwards the caller's provider credentials, or it holds provider credentials itself and authenticates callers separately. NIST notes that some proxy agents can authenticate each individual user, which suggests the second model, but it has consequences for key management and for how evidence identifies the calling agent.

Finally, check retries and timeouts. Agents and provider clients retry on failure, and the proxy adds its own latency and its own timeout. Mismatched settings can produce duplicate requests, duplicate evidence records or silent failures. Align timeouts across the agent, proxy and provider, and decide whether a retry counts as a new enforcement decision.

Verifying that no call bypasses the proxy

Configuration-level routing is advisory until the network enforces it. A team can reset an endpoint, add a second client, or deploy an agent from an old template, and calls will quietly go direct. The matching control for the proxy model is an egress rule that blocks direct connections to provider endpoints from workloads, so that the proxy is the only host permitted to reach them.

Verify from both sides. On the proxy side, compare the set of agents and workloads that appear in proxy records with your inventory, and investigate any that are missing. On the network side, review egress deny events and flow data for attempts to reach provider addresses directly. A workload that is active but absent from proxy records, or one generating denied egress attempts, points to a bypass path. Run these checks continuously rather than once at cutover, since new agents appear regularly.

The proxy's records should be built for audit use. At a minimum, each record should capture a timestamp, the calling workload or agent identity, the target model, the policy version applied, the decision (allow, redact or block) and the reason. Whether and how to record request and response content depends on your data handling and retention requirements.

Runtime enforcement, evidence and the limits of this approach

Trussed enforces policy in the execution path, so decisions on model calls and tool calls are enforced at runtime, not reviewed after the fact. Policy can allow, redact or block an interaction, and can also rate limit or escalate for human approval. Each enforcement decision generates audit-ready evidence, mapped to frameworks including HIPAA, NAIC, NYDFS, the EU AI Act and NIST. Trussed can be deployed self-managed in the customer cloud, with configurable retention and data residency, which matters when prompts and responses cannot leave your environment.

The approach has limits. It governs what passes through the proxy, so a call that avoids the proxy is not covered, which is why egress controls are part of the design and not an optional extra. Agents that hardcode their own endpoints or build their own clients may need code changes or network-level routing. The proxy also becomes a dependency on the model-call path, so availability, latency and failure behavior need the same planning as any other critical gateway. Finally, enforcement at the level of individual interactions and tool calls is not the same as tracing an entire agentic workflow or governing a business transaction across agents. Plan separately for those needs.