How to Replay Production LLM Traffic Against a New Model Version Before Cutover at a Bank

What changed in the supervisory frame

Many bank procedures still cite OCC Bulletin 2011-12 as the basis for model change validation. On April 17, 2026, the OCC, Federal Reserve Board and FDIC issued updated interagency model risk management guidance that rescinds prior guidance. The OCC is rescinding Bulletin 2011-12, Bulletin 2021-19, Bulletin 1997-24 and the Comptroller's Handbook booklet on model risk management. The updated guidance covers model development and use, validation and monitoring, governance and controls, and vendor and third-party products.

It also states that practices should be risk-based and commensurate with the size, complexity and extent of model use, and that it is most useful for models supporting significant business lines, operations, services and functions. For a platform team, this means replay depth should scale with how consequential the use case is. A customer-facing assistant that touches account information warrants a larger and more carefully documented replay than an internal drafting tool. Update your change-control references so that your procedures cite the current guidance, and confirm with your model risk function how a vendor LLM version swap is classified in your inventory.

Step 1: Capture, sample and sanitize the traffic

Start with the question the replay must answer: will the candidate behave acceptably on the workloads the incumbent serves today? Sample to answer it. Stratify by use case, intent, language, conversation length and tool usage, and include rare but consequential cases such as complaints, disputes and requests that trigger policy blocks. Use a time window long enough to cover seasonal patterns, and hold out a later window as an out-of-time set so you are not tuning prompts against the same data you use for acceptance.

Sanitize before anything is stored for replay, not after. Production prompts and responses can contain names, account numbers and other regulated data, and OWASP's guidance on sensitive information disclosure points to data sanitization and strict input validation as core controls. In practice, replace identifiers with typed, consistent tokens so that a customer referenced three times in a conversation remains one entity, and exclude record categories that cannot be reliably redacted. Test the redaction on a labeled sample and record its miss rate, because redaction that changes meaning can distort the comparison.

Treat the sanitized dataset as a controlled asset. Restrict access to named roles, set a retention period approved by your privacy and records functions, and keep the data in the same residency boundary as production. Record the sampling method, window and exclusions as part of the dataset definition.

Step 2: Replay without customer impact and measure the noise floor

Shadow the candidate so its outputs are never returned to users and it cannot cause side effects. For multi-turn conversations, replay whole conversations with the recorded earlier turns as context. For tool calls, serve recorded tool results from a stub rather than calling live systems, and note where the candidate requests a call the incumbent did not make, since that divergence is itself a finding.

Non-determinism makes a single pairwise diff unreliable. A comment letter to the Federal Reserve describes a controlled financial classification study of two vendor model migrations that handled this directly. Incumbent and successor received identical dated inputs with matched sampling seeds, and each version ran three replicates across 12 analysis dates, for 14,400 calls. The authors estimated a noise floor from disagreement between replicates of the same version, then treated the cross-version flip rate in excess of that floor as the primary endpoint. This is a commenter's proposal and one study, not agency guidance, but the design transfers well: run the incumbent against itself, and attribute only the excess difference to the new version.

Record the full configuration for every paired run, including inputs, seeds where the provider allows them, replicate count, dates and prompt and parameter versions, so a reviewer can reproduce the comparison.

Step 3: Compare outputs and policy outcomes

Choose metrics that fit the task. For classification-style outputs, the same study reports excess flip rate, label-distribution shift, per-entity churn and directional drift. Overall agreement can look healthy while a distribution shift or a one-directional drift changes decisions, so report both aggregate and per-entity views. For free-text generation, define task-specific checks instead of string diffs: structured-output validity, presence of required disclosures, refusal and escalation rates, and factual consistency against the provided context. Add human review of a stratified sample, scored with a written rubric, for the cases automated checks cannot judge.

Compare policy enforcement separately from answer quality. A new version may produce text that trips a different rule, request tools differently, or leak a field that the incumbent did not. For each replayed interaction, record the allow, redact or block decision and the rule that produced it, then tabulate decision agreement between versions. Investigate every case where the candidate moves from block or redact to allow, since that direction carries the most risk. Pair this with adversarial and sensitive-information testing before cutover, for example using the four areas in OWASP's red teaming guide: model evaluation, implementation testing, infrastructure assessment and runtime behavior analysis.

Evidence for model risk review, and where runtime enforcement fits

Reviewers and examiners will want to inspect what was tested, how, and who decided. Retain the dataset definition and sanitization method, the run configuration, noise-floor estimate, metric results with thresholds, the policy decision comparison, exceptions with their dispositions, and the cutover approval. Keep it in a form that can be reproduced and retrieved on request, and continue monitoring after cutover, because risk management for LLMs does not end at deployment.

Trussed does not run the replay or the model comparison. Its role is narrower: it enforces policy in the execution path, with allow, redact and block decisions applied at runtime, so the same policy applies to live and replayed interactions. That gives the policy comparison in step 3 a consistent basis, and the enforcement itself produces audit-ready evidence of what was allowed, redacted or blocked. Trussed deploys self-managed in the customer cloud, with configurable retention and data residency, which bears on where replay-related evidence is held. Trussed is deepest in insurance, a regulated financial sector, and makes no claim of bank deployments.