Enforcing data residency per request in a LiteLLM Proxy

Residency is a per-request decision, not a deployment setting

A shared LiteLLM Proxy sits in front of many teams and applications. Each of them may carry a different residency obligation, and the proxy chooses a concrete model endpoint for every call. That choice is where residency is either kept or lost.

Static regional deployments and per-team configuration set intent, but they do not constrain the individual routing decision. A request can still reach an out-of-region model through a misconfigured model group, a fallback list that names a deployment elsewhere, a retry that lands on a different instance, or a request that arrives without any residency attribute and gets routed by default.

The general architecture that fits this problem is the one security guidance describes for proxies: a Policy Enforcement Point that allows or denies each request based on a verdict from a decision function, using context evaluated on every request. Applied here, the decision is whether the request is permitted and in which region. Enforcement is routing to a compliant deployment or rejecting the call.

Identify the residency requirement from trusted context

Start by deciding where the requirement comes from. The usual candidates are the virtual key, the team the key belongs to, and request metadata or a header supplied by the calling application. They differ in trust. A key or team attribute is set by the platform and cannot be changed by the caller. A header or metadata field is set by the caller and can be wrong or absent.

A workable precedence is to treat the key or team attribute as authoritative and accept request-level metadata only as a narrowing signal, for example a request for one region when the team is permitted two. If the request asks for a region the key is not permitted to use, treat that as a conflict, not a hint.

Define the missing and conflicting cases explicitly before writing any routing configuration. For regulated workloads, default-deny is the safer choice: if no residency requirement can be resolved, reject the request with a clear error instead of picking a default region. The same applies when the requested region has no available deployment. Validate the attribute's presence and allowed values before routing, in the way API schema validation rejects malformed input before business logic runs. A custom hook or callback that runs before routing is a natural place for this check.

Restrict routing to region-compliant deployments

The proxy's native building blocks are model groups, routing and fallback settings, virtual keys and teams, request metadata, and hooks or callbacks. Used together, they can express a region boundary. The aim is that after the residency decision, the set of candidate deployments contains only in-region endpoints.

• Separate model groups by region: Expose region-specific model names, one group per region, so a group never mixes deployments from different geographies.
• Bind keys and teams to permitted groups: Restrict each virtual key or team to the model groups of its permitted regions, so the platform sets the boundary instead of the caller.
• Resolve and validate in a pre-routing hook: Read the residency attribute, apply the precedence rules, and map it to a region-scoped model group before the router selects a deployment.
• Reject instead of defaulting: Return an explicit error when the attribute is missing, invalid, or has no matching deployment.
• Test the boundary: Send requests with missing, conflicting, and unsupported regions in a non-production environment and confirm each is rejected or routed in region.

Stop fallbacks, retries, and load balancing from crossing regions

Retries, failover, and load balancing are standard proxy resilience features, and they are designed to find another instance when the first one fails. That is the reason they need a residency constraint. A request correctly routed to an in-region deployment can still be re-sent elsewhere when that deployment times out or is rate limited.

Three patterns reduce the risk. First, keep each fallback list inside one region, so a failure in a region-specific group falls back only to another group in that same region. Second, never configure a global or default fallback that spans regions, because it applies to requests whose residency was already decided. Third, keep load-balanced deployments within a model group in the same geography, so balancing cannot select a distant instance.

The tradeoff is availability. A region-contained fallback chain has fewer options, and a regional outage can mean the request fails. For a residency-bound workload, a failed request is the correct outcome; the alternative is a silent compliance breach. Make that behavior explicit to application owners, and surface it as a distinct error so they can retry later or alert.

Where native configuration stops and runtime enforcement starts

Proxy-native configuration is a sound baseline, and a team that never adds anything else can still build most of what is described above. Its limits are operational. Rules are spread across model groups, key and team settings, fallback lists, and hook code, so a single edit can reopen a cross-region path. Evidence is whatever the hook and callbacks emit, and its completeness depends on each team writing it correctly. Policy changes to routing take effect in the same place that serves production traffic, which makes versioning and testing harder.

Trussed addresses this layer. It enforces policy in the execution path, on individual interactions and tool calls, with outcomes of allow, redact, or block, enforced at runtime, not reviewed after the fact. Enforcement itself generates audit-ready evidence for each request, mapped to frameworks that include HIPAA, NYDFS, the EU AI Act, and NIST. For teams that need to keep the control and the evidence within their own boundary, Trussed is self-managed in the customer cloud, with configurable retention and data residency.

In practice, this means the residency rule is expressed once as policy and evaluated on every request, alongside the proxy's own routing, instead of being reconstructed from configuration after an incident.