Proving that routing to a cheaper LLM did not degrade regulated outputs

Define non-degradation before the change

Most teams approach this as a model-quality question. An auditor will approach it as an evidence question: what did you say would stay true, and what can you show?

A workable definition has two parts. For each regulated output type, the replacement model must stay within the same policy limits as the incumbent, and it must stay within a pre-agreed quality margin relative to the incumbent. Policy limits might cover handling of regulated data in responses, permitted actions in tool calls and required output format. The quality margin is a number someone with authority approved, with a method for measuring it, before cutover. This definition is a design choice, not a regulatory standard.

The regulatory texts set a general frame. As of October 2026, the consolidated EU AI Act text requires high-risk AI systems to technically allow automatic recording of events over the lifetime of the system, and requires a documented, maintained risk management system. The NIST AI RMF is voluntary and can structure the work, but it is not a legal obligation. None of these hands you a routing evidence template, so the architect has to design one that a reviewer can follow.

The five categories of evidence

A defensible claim rests on several kinds of evidence, each answering a different question. No single category is sufficient on its own.

Pre-routing baselines come first and must be captured on the incumbent before anything changes. Without them, post-change numbers have nothing to be compared against.

Per-interaction policy enforcement records show what the policy permitted and what happened on live traffic. Comparative output sampling addresses quality. A single accuracy figure hides the composition of the test set. A February 2026 NIST report notes that evaluators often report one accuracy metric and describes generalized linear mixed models that estimate item difficulty alongside model capability. The practical lesson is to compare models with a method that controls for task mix rather than raw pass rates.

Human escalation outcomes show whether reviewers upheld or overturned what the system flagged. Retention determines whether any of this still exists when the question arrives.

• Pre-routing baselines: Violation, redaction, block and escalation rates, plus sampled quality, on the incumbent model.
• Per-interaction enforcement records: Policy applied and action taken on each live interaction and tool call.
• Comparative output sampling: Pre-defined sampling, margin and difficulty-aware comparison method.
• Human escalation outcomes: Reviewer decisions on escalated items, comparable across both periods.
• Retention: Documented periods and form (content or metadata) for each evidence class.

Which evidence comes from where

Keep runtime enforcement records and offline evaluation artifacts as separate, cross-referenced evidence classes. Presenting enforcement records as proof of output quality will not survive scrutiny.

• Which policy version applied, and what action was taken?: Runtime enforcement records on each interaction.
• Which model did the routing decision select?: Runtime record of the routing decision, tied to the same interaction.
• Are redaction and block rates comparable before and after?: Runtime records, compared against the incumbent baseline.
• Is output quality within the agreed margin?: Separate offline evaluation and human review of sampled outputs.
• Did task mix shift between periods?: Separate statistical analysis, outside runtime enforcement.
• Did reviewers agree with escalations?: Human review process, with outcomes recorded and linked to the interaction.

What a per-interaction record needs to contain

For an auditor to verify that policy was enforced equivalently across models, each interaction needs a record that joins the routing decision to the enforcement outcome. A candidate field set is the routing decision, the policy version, the enforcement action (allow, redact, block or escalate), what was redacted, the escalation outcome and the retention class. This field list is a practitioner design, not a mandated schema.

Two design choices make comparison easier. First, apply the same policy and the same policy version to both models. If the policy differs between periods, a change in outcomes cannot be attributed to the model. Second, prefer structured outputs with schema validation, and enforce allowed-action schemas for tool calls, as OWASP RAG security guidance recommends. Machine-checkable output shapes make before-and-after comparison tractable.

Retention needs a deliberate decision. OWASP guidance advises against logging raw queries, retrieved content, outputs or tool arguments by default, because they may hold secrets or sensitive personal data. Logging duties for high-risk systems under the EU AI Act pull the other way. A common resolution is to retain enforcement metadata for the full audit period and retain content only where a documented need and control exist. Write the decision down for each evidence class.

Structure the routing change so the evidence holds

Treat the routing change as a controlled change under your risk management process, with documented approval, acceptance criteria and rollback conditions. The sequence below keeps each claim backed by something recorded before the fact.

• Write the non-degradation definition, quality margin and sampling plan, and obtain approval before cutover.
• Capture baselines on the incumbent: policy violation, redaction, block and escalation rates, and sampled quality on the regulated task.
• Confirm the same policy version is enforced on both models and that the routing decision is recorded on every interaction.
• Route a share of traffic or selected workflows first, then compare enforcement records and sampled quality against the baseline.
• Widen only if criteria are met, and record reviewer outcomes on escalated items for both periods.
• Archive the plan, comparisons and approvals alongside the interaction records under the agreed retention schedule.

Where runtime policy enforcement fits

Trussed enforces policy in the execution path across models, agents, tools and MCP servers, with actions to allow, redact, block, rate limit or escalate for human approval. The scope is individual interactions and tool calls. Because enforcement happens at runtime, not reviewed after the fact, each action produces audit-ready evidence as a by-product. With 100+ models under one policy plane, the same policy can apply before and after a routing change, which makes the records comparable. Evidence is mapped to HIPAA, NAIC, NYDFS, the EU AI Act and NIST, and deployment is self-managed in the customer cloud with configurable retention and data residency.

The boundaries matter. Trussed does not run offline evaluation, set your quality margin or judge output quality, so those remain separate processes. It also does not provide model lineage or end-to-end tracing of agentic workflows. What it contributes to this evidence model is the per-interaction record of what the policy allowed and what was done.