Fail-Open or Fail-Closed: How to Handle Detector Timeouts in LiteLLM Guardrail Hooks

Where detector timeouts surface in guardrail hooks

A guardrail hook that calls an external detector adds a network dependency to the request path. The hook runs at up to three points: before the model call (pre-call), alongside it (during-call), and after the response is produced (post-call). At each point the detector can be slow, return an error, or never answer. The hook code has to turn that condition into a decision, either letting the request continue or raising an exception that rejects it. If the implementation does not handle the condition explicitly, the outcome is whatever the surrounding code and defaults happen to do, and that is the implicit choice this guide argues against.

The exposure differs by stage. A pre-call failure that fails open sends an unscreened prompt to the model, which is the situation prompt-injection screening exists to prevent. A post-call failure that fails open releases an unscreened response to the user or to downstream code, which is the output-handling risk OWASP lists in its 2025 LLM Top 10. A check that runs concurrently with the model request adds a further question: whether the model call is allowed to complete, or is abandoned, when the verdict never arrives.

OWASP's prompt injection guidance also makes a point that applies here: guardrail and approval hooks are application configuration, and their presence does not establish that an action is authorized. A hook that silently passes traffic when its detector is down is still a configured hook, and it still looks healthy on a diagram.

Failure policies compared

Each policy trades availability against exposure. The table describes general patterns, not LiteLLM settings.

• Fail-open: Request proceeds without a verdict. Highest availability, highest exposure. Acceptable only where the data and action are low risk, and only if the bypass is recorded.
• Fail-open with flagging: Request proceeds, is marked as unscreened, and is queued for review or later scanning. Keeps traffic flowing and preserves a record, but cannot recall a response already delivered.
• Fail-closed: Request is rejected when no verdict arrives. Lowest exposure. A slow or unavailable detector becomes an outage for that route.
• Tiered or conditional: Failure mode is set per route, data class or action type. Requires a maintained route inventory, but matches the control to the risk.
• Bounded retry: One or two retries inside the total latency budget. Absorbs transient faults, but each retry spends time that the request may not have.
• Secondary detector fallback: A second, usually simpler, detector issues the verdict when the first fails. Narrows the gap, but its coverage is different and must be stated in the policy.

A decision framework: sensitivity, action type and route

The matrix below is a working framework for platform teams, not a published standard. Start from three inputs for each route.

Data sensitivity. Routes that carry regulated personal data, credentials or confidential records should default to fail-closed for PII and similar detectors, because the cost of one unscreened request is not recoverable. Routes handling public or low-sensitivity content can tolerate fail-open with flagging.

Action type. A request that only generates text for a person to read differs from one that triggers a tool call, writes to a system, or sends data outward. Where the model's output can cause a change, a missing verdict should block or escalate for human approval rather than pass. OWASP's guidance to specify a safe failure path, with a stop condition, when a dependency is unreachable supports this reasoning.

Route and detector. Pre-call prompt-injection checks on routes with tool access usually warrant fail-closed. Post-call content classification on an internal summarization route may justify fail-open with flagging. Document the choice per route and per detector, because one route can need different modes for different detectors.

A practical default is to assign every route to a tier, with fail-closed as the tier for anything not yet classified. Relaxing a route to fail-open then becomes a recorded, reviewed decision instead of an accident.

Implementing explicit timeout and exception handling

Put the policy in the hook code, where it can be read and tested, rather than relying on inherited behavior. The pattern has five parts.

First, wrap every detector call in its own timeout, shorter than the route's overall request timeout. OWASP recommends timeouts for resource-intensive operations in general; the detector-specific value should come from measured latency, for example a high percentile of observed detector response time plus headroom, not a guess.

Second, catch timeouts and errors separately from a genuine detector verdict. A timeout, a connection error and a malformed response are all "no verdict", and none of them should be treated as "clean".

Third, look up the route's failure mode and apply it: return the request unchanged and flag it, raise an exception that rejects it, or hand it to an escalation path.

Fourth, cap retries. A retry is useful for transient faults only if the total of attempts still fits inside the budget. Retries on a saturated detector can deepen the outage, so combine them with a circuit-breaker style rule that stops calling a detector after repeated failures and applies the failure mode directly.

Fifth, emit a decision record every time the failure path is taken, so the degraded period can be reconstructed afterward.

• Per-detector timeout set below the route timeout
• Timeouts, errors and malformed responses treated as no verdict
• Failure mode looked up per route and detector
• Retry limit inside the total latency budget, with a stop rule after repeated failures
• Decision record written on every failure-path outcome

Validating the failure paths before production needs them

Test the degraded path as deliberately as the healthy one. Inject faults by delaying detector responses past the timeout, returning errors, and dropping connections, then confirm each route resolves to its documented mode and that a decision record is written. Run load tests that push detector latency toward the budget, since a detector that passes at low volume can time out under production concurrency. Retain the test results as evidence, so the configuration is shown to behave as designed rather than assumed to.

In production, track detector latency percentiles and the rate of timeouts and errors per route, and alert when the share of requests resolved without a verdict rises. A rising count of fail-open passes on a sensitive route is a signal to review the tier, not just the detector.

Extending the pattern with enforcement in the execution path

The approach above can be built inside a single gateway deployment. It becomes harder to sustain across many models, teams and routes, where each hook can drift into its own failure behavior and its own record format.

Trussed enforces policy in the execution path, so the outcome for a detector-unavailable state is defined in advance and enforced at runtime, not reviewed after the fact. The defined outcome can be allow, redact, block, rate limit, or escalate for human approval, applied to individual interactions and tool calls. Each enforcement decision generates audit-ready evidence, mapped to frameworks including the EU AI Act and NIST, so a degraded-detector period can be shown to an auditor as recorded decisions rather than reconstructed from application logs.

Trussed has run in continuous production in a regulated enterprise for more than 18 months and supports 100+ models under one policy plane. The design principle is the same as in this guide: the failure decision belongs to policy, and the evidence belongs to the decision.