What happens to in-flight LLM requests when an AI governance proxy pod restarts
Requests in flight on a terminating proxy pod survive only if the proxy stops taking new work, finishes or cleanly ends active requests, and exits within the termination grace period. Otherwise unary calls fail, streams are truncated, and tool calls may be half executed. Policy decisions and audit records can also go missing. Graceful shutdown, readiness gating, idempotent retries, durable audit writes, and a deliberate fail-open or fail-closed choice reduce these gaps.
Why proxy termination is different for LLM traffic
A governance proxy sits in the request path, so a pod restart affects more than the connection. It can interrupt the request itself, the policy decision applied to it, and the audit record that describes it.
A request on a terminating pod survives only if the proxy stops taking new work, finishes or cleanly ends active requests, and exits within the termination grace period. What happens next depends on how far the shutdown sequence gets before the pod is removed.
Proxy pod termination, in order
| Stage | What happens |
|---|---|
| 1. Pod marked terminating | Endpoint removal and shutdown begin, but new requests can still arrive for a short time. |
| 2. Readiness fails, draining starts | The proxy stops accepting new work and lets active requests finish. |
| 3. Grace period expires | Requests still running, such as long streams, are at risk. |
| 4. Forced kill | Connections reset, and unflushed policy and audit state is lost. |
What each request type experiences
Unary, streaming, and tool-calling requests fail in different ways. The outcome depends on whether the request finishes, or is cleanly ended, before the grace period expires.
| Request type | If it finishes or ends cleanly in time | If the pod is force-killed |
|---|---|---|
| Unary | The call completes normally. | The call fails. |
| Streaming | The stream completes or is ended cleanly. | The stream is truncated when the connection resets. Long streams are the most exposed. |
| Tool-calling | The tool call completes. | The tool call may be half executed. A client retry can duplicate execution unless idempotency keys or deduplication prevent it. |
Where the governance gaps appear
Beyond failed requests, termination can leave gaps in enforcement and evidence:
- Late arrivals: New requests can still reach the pod for a short time after it is marked terminating.
- Lost state: Policy and audit state that has not been flushed is lost when the pod is force-killed.
- Audit integrity: If the decision record is not written durably before a request is forwarded, a kill can leave a request without its record.
- Duplicate tool calls: A retried tool call can execute twice without idempotency keys or deduplication.
- Invisible failure modes: If a fail-open or fail-closed outcome is not recorded and surfaced, the gap goes unnoticed.
Design mechanisms that reduce disruption
- Handle SIGTERM as a drain signal: On SIGTERM, stop accepting new connections, let active requests complete, flush pending records, then exit.
- Gate readiness: Fail the readiness probe as soon as draining starts so the pod stops receiving new traffic.
- Use a preStop delay: A brief pause lets endpoint removal propagate before shutdown begins.
- Size the grace period from real data: Set terminationGracePeriodSeconds above the longest request you intend to protect, and cap stream duration deliberately.
- Limit voluntary disruption: Run multiple replicas and use a PodDisruptionBudget so drains and upgrades never remove all capacity at once.
- Write audit records durably: Persist the decision record before forwarding where possible, and buffer to durable storage so a kill does not erase it.
Fail-open or fail-closed during disruption
When enforcement is unavailable, traffic either continues without enforcement (fail-open) or is blocked (fail-closed). This should be a deliberate choice rather than an accident of how the proxy shuts down.
Where possible, set the behavior per policy or route, and record and surface each failure mode so that gaps can be reviewed afterward.
Plan for restarts before they happen
If you are assessing how runtime enforcement behaves under pod disruption, we can walk through these criteria with your platform team.
Discuss restart behavior with an expert