Out-of-Distribution Input in AI: Definition, Runtime Risk, and Enforcement at the Interaction Level

What an out-of-distribution input is

OOD is a property of an input relative to the data a model was trained or evaluated on. NIST's adversarial machine learning terminology describes OOD data as data collected at a different time, and possibly under different conditions or in a different environment, than the training data. A recent peer-reviewed review phrases it as samples outside the support covered by training data, arising from new devices, environments, classes or abnormal operating conditions. Neither definition implies malicious intent.

For classical models, this is often easy to picture: a sensor type the classifier never saw, or a class missing from training. For LLMs and agents, the boundary is harder to draw. Prompts, retrieved documents, tool outputs and conversation history combine in ways no evaluation set fully covers. An input can look ordinary to a human reader and still sit far from anything the system was tested against.

The practical consequence is that the model's behavior on such an input is unvalidated. It may produce an unreliable answer, or, when the system can act, an unintended action.

OOD compared with related concepts

These terms overlap in practice, which is why they are often conflated. Separating them matters because the cause shapes the response, even though the same score may rise for all of them.

• Out-of-distribution input: An input outside the training or evaluation data. Defined by its relationship to data, not by intent.
• Distribution shift or drift: A change in data over time or conditions. Persistent shift makes many inputs OOD relative to the original data.
• Adversarial input: An input deliberately optimized by an attacker to cause failure. NIST notes it may come from the training distribution or be OOD.
• Prompt injection: Modifying the input given to a generative system so it behaves in unintended ways. It manipulates instructions or context, and direct injections can affect interconnected systems downstream.
• Mismatched generalization: A class of LLM attack using inputs outside the model's safety training distribution but within its capabilities training, so the model still understands them.

Detection approaches and their limits

Detection methods differ mainly in what they examine. Confidence-based methods look at the model's predicted probabilities. Distance-based and embedding-based methods measure how far an input sits from known data in a feature or latent space. Reconstruction-based methods look at residual error. The recent review surveys detectors that use raw inputs, latent representations, probability distributions, reconstruction residuals, temporal relationships and protocol behavior, though its focus is IoT sensors rather than LLMs.

The limits are structural. NIST cautions that detection is impossible when the in-distribution and OOD distributions overlap. Any detector score is therefore probabilistic, and a threshold trades missed detections against false alarms. The same review notes that natural drift, OOD inputs and adversarial examples can all produce high suspiciousness scores, so a score alone does not say why an input is unusual.

For LLMs, NIST describes classifier-based screening of inputs and outputs as a possible added layer of defense, but notes those classifiers are attackable and may fail in correlated ways with the models they monitor. Interpretability-based anomaly detection and keyword or perplexity defenses are described as early work. Monitoring OOD behavior of deployed agents is listed as an open challenge. A detector is a useful signal, but it cannot be the only control.

Runtime risk when the system can act

A chat model that receives an unfamiliar input can return a poor answer. An agent with tool access can do more: call a tool with malformed or unexpected arguments, retrieve data it should not, or send output to an interconnected system. This is architectural reasoning rather than a documented incident class, but it follows from how tool-calling systems work. The model's output becomes an action, and the model's behavior on the input was never validated.

Pre-deployment testing cannot close this gap, because the defining feature of an OOD input is that testing did not anticipate it. Detection narrows the gap but, as above, does not eliminate it. What remains is a decision: given an uncertain input and a requested action, should the action proceed?

That decision is best tied to the risk of the action, not to a diagnosis of the input. In the moment, the system usually cannot tell whether an unusual input is benign drift, an adversarial example or an injection. A read-only lookup and a write to a production record warrant different treatment under the same uncertainty.

Enforcing policy at the interaction and tool-call level

Enforcement in the execution path means the decision is made before a model response is released or a tool call executes, not reviewed afterward. The outcomes below are common policy actions. The mapping to OOD cases is design guidance: NIST-described responses to suspicious inputs include rejection, deferral and escalated review, and the AI RMF Playbook lists throttling among its security countermeasures. No source prescribes thresholds, so they must be set per organization.

• Allow: Low-risk actions proceed when the signal is weak or absent. The decision is still recorded.
• Redact: Remove sensitive fields from a prompt, response or tool payload when the content is the risk, not the action itself.
• Block: Stop high-risk tool calls or responses when the signal is strong or the action is irreversible.
• Rate limit: Slow repeated unusual requests, which limits the damage of automated probing or runaway loops.
• Escalate for human approval: Hold consequential actions for review. Define who approves, how long the request waits, and what happens on timeout.