How Model Validators Test Nondeterministic Tool Selection in LangGraph Agents

Why tool selection varies between runs

In a LangGraph agent, the model often decides what happens next. A tool-calling step presents the model with a set of tool schemas and the conversation state, and the model returns which tool to call and with what arguments. When routing depends on that output, the graph's path is only as repeatable as the model's decision.

Several factors contribute. Sampling settings such as temperature change how much the model's output varies, and lowering them reduces variation without guaranteeing identical output. Providers can serve the same model with different capabilities, such as context length or tool-call support, so NIST guidance on benchmark evaluations treats provider choice as a scaffolding setting. Prompt wording and tool descriptions shape which tool looks most relevant. Accumulated state, including earlier tool outputs, changes the context for each later decision.

Content the agent reads is another source. NIST-hosted material notes that agents pick tools without strong constraints and that ambiguous or injected content can influence both tool choice and arguments. This variability is not ordinary sampling noise, so it needs its own test cases.

Some factors can be constrained: the available tool set, schema strictness, sampling settings and the pinned model and provider configuration. Others, including provider-side variation and content-driven influence, can only be measured and bounded.

Defining metrics and acceptable boundaries

Validators need metrics that map to approved behavior. Common choices follow from the data collected above. Tool-selection consistency is the share of runs for each input that choose each tool, reported with an uncertainty interval. Sequence variance counts the distinct call orders observed per input and how often each occurs. Argument conformance is the share of calls whose arguments pass schema and value checks. Forbidden-call rate is the share of runs that invoke a tool outside the approved set for that task.

Thresholds are a governance decision, not a technical default. A reasonable pattern is to define, per task type, the set of permitted tools, the minimum acceptable consistency for the expected tool, and a tolerance for forbidden calls that is stated explicitly. For high-consequence tools, the tolerance may be set to zero observed events, with the understanding that zero observed in a sample is not zero probability.

Test evidence should also cover multi-step sequences. Legitimate tools can be chained in harmful ways that remain within permissions, so checking each call in isolation can miss sequence-level risk.

What sampled validation can and cannot show

Sampled validation can establish how frequently an agent selects a given tool, how stable that rate is across settings, and how often argument checks fail under defined conditions. It supports a statistical conclusion that the agent behaves within bounds under the tested configuration and inputs.

It cannot guarantee the outcome of an individual production interaction. Production inputs will fall outside the test set, configurations change, and injected content can steer a specific call. A low failure rate in testing still leaves a nonzero chance on any given call.

NIST AI RMF frames testing, evaluation, verification and validation as continuing through deployment and operations, including ongoing monitoring, periodic testing and incident tracking. Pre-deployment validation is therefore one part of a lifecycle. Teams with EU AI Act exposure should also confirm current application dates. As of November 2026, the Commission's Digital Omnibus on AI proposes delaying high-risk rules to 2 December 2027 for stand-alone systems and 2 August 2028 for systems embedded in products, and adoption rests with the legislator.

Which failure modes each control addresses

Validators should state, for each tool-use failure mode, what sampling can demonstrate and what must be enforced on the call itself.

• Wrong-tool selection: Sampling: estimates frequency per input. Runtime: restrict the tools permitted for a given context.
• Unauthorized tool invocation: Sampling: detects occurrences in tested cases. Runtime: block calls outside approved scope on every call.
• Malformed or out-of-scope arguments: Sampling: measures conformance rate. Runtime: validate and allowlist parameters at the tool boundary regardless of the model's choice.
• Prompt-injection-induced calls: Sampling: probes with adversarial cases. Runtime: apply policy to each call, with human approval for sensitive actions.
• Excessive call volume: Sampling: observes typical volumes. Runtime: rate limit calls.

Where runtime enforcement fits

Controls at the tool boundary do not depend on which tool the model selected. Allowlisting parameters, sandboxing tool execution and recording every invocation with exact parameters and returned output give validators and auditors a record that does not rely on the model behaving as sampled.

Trussed is a runtime control that enforces policy in the execution path across agents, tools and MCP servers. Each individual interaction or tool call can be allowed, redacted, blocked, rate limited or escalated for human approval. Enforcement is applied at runtime, not reviewed after the fact, so it addresses the residual risk that sampling leaves on a single call.

The enforcement generates audit-ready evidence, mapped to frameworks including NIST and the EU AI Act, showing what a call was permitted to do and what happened. Trussed does not replace pre-deployment validation. It complements measured, bounded results with enforced limits on individual tool calls, and it can be deployed self-managed in the customer cloud with configurable retention and data residency.