AI Bias Testing Documentation Template: What to Record per Test Run for NAIC Exams

Why the test run is the unit of documentation

The NAIC Model Bulletin expects insurers to develop, implement and maintain a written AIS Program for the responsible use of AI systems. That program includes documentation of validation, testing and auditing, including evaluation of model drift. The bulletin also encourages verification and testing methods to identify errors and bias, and the potential for unfair discrimination in resulting decisions and outcomes.

A program document describes intent. An examiner asking for evidence wants specific events: a test, on a date, against a defined population, with a recorded result and a decision about what to do next. When those events live in scattered spreadsheets, notebooks and tickets, the same fields tend to go missing: scope, data used, thresholds, remediation and approval. A single record structure applied to every run closes that gap.

The template below is split in two parts. The first covers what was tested and why. The second covers how it was measured and how the run was closed. Each field notes why an examiner is likely to ask for it. Bias testing itself happens in your own testing environment or with your vendors. The record is how you prove it happened.

Part 1: Identification, context and scope

These fields establish which system was tested and under what conditions. Without them, a result cannot be tied to a production system or a decision.

• Record ID and AI system identification: Unique run ID, system name, version and owner. Examiners need to match a test to the system in production.
• Use case and decision context: Whether the system supports underwriting, claims, pricing or customer interaction, and what decision or outcome it influences. Materiality depends on this.
• Objective and trigger: What the run was meant to detect, and why it ran: pre-production, scheduled, after a material change, or annual output and drift testing. Link the change ticket where one exists.
• Model type and origin: Predictive or generative, and internal or vendor-supplied. For vendor systems, record which vendor testing you relied on. The NAIC evaluation tool distinguishes internal and third-party systems.
• Data and population definition: Datasets, time window, filters, sample size and how the tested population was selected. Record whether the data is suitable for the test, since the bulletin addresses suitability of data used to develop, train, validate and audit.
• Protected-class or proxy methodology: How protected-class status was established or inferred, which proxies were used, and the reasoning for that choice. Examiners ask how you tested when direct attributes were unavailable.

Part 2: Metrics, results and closure

These fields show what was measured, what happened and who accepted the outcome. They are the ones most often missing from informal records.

• Metrics and thresholds: Metric definitions, the threshold for each, and why it was chosen. The NAIC evaluation tool asks for testing methodology, and NYDFS expects documented reasoning behind it.
• Results: Computed values per metric, per group, with the run date, tool and code version. Enough detail that another analyst could reproduce the run.
• Exceptions: Any metric outside threshold, any group with insufficient data, and any deviation from the planned method. Record deviations even when the overall result passed.
• Remediation: Actions taken or accepted as risk, owner, due date and the follow-up run that verified the fix. Examiners look for the loop from finding to closure.
• Reviewer sign-off: Name, role, date and decision of the independent reviewer or approver. Keep the author and approver distinct.
• Retention and linkage: Retention period, storage location and links to AIS Program policies, related runs and change tickets. Compliance and counsel should set the retention period.

How the fields map to NAIC, NYDFS and Colorado

The NAIC Model Bulletin lists documentation of validation, testing and auditing, and notes that the nature of that work should reflect the underlying components of the AI system. That is the reason the template records model type. The NAIC AI Systems Evaluation Tool asks insurers to describe testing frequency, scope and methodology, and it covers testing of model outputs for drift, accuracy, bias, unfair trade practices and performance degradation. The AI Risk Evaluation Supplement asks how outputs were tested for unfair discrimination, how the model was validated before deployment, and how performance is monitored on an ongoing basis. Objective, trigger, scope, metrics and results answer these directly.

NYDFS Insurance Circular Letter 7 (2024) adds two points. An insurer should document the processes and reasoning behind its testing methodologies for unfair or unlawful discrimination, commensurate with the complexity and materiality of its AI use, and be prepared to provide that documentation to the Department on request. It also says testing should occur before production, on a regular cadence, and after material updates or changes, with testing at least annually to assess model output, including drift. The trigger field and the rationale behind metric, threshold and proxy choices cover this.

Colorado has changed. As of October 2026, the Colorado Attorney General reports that SB 26-189, signed in May 2026, repeals and reenacts the SB 24-205 provisions with new requirements for automated decision-making technology in consequential decisions. Do not build fields from the earlier SB 24-205 duties. Add a column for Colorado after you review the new law and rulemaking, and assign an owner to track it. Colorado's 3 CCR 702-10, listed as implementing the NAIC bulletin, is a separate item to review with counsel.

A mapping column in the template, naming the NAIC, NYDFS and any state reference for each field, lets one record serve several regimes.

Supplementing test records with runtime enforcement evidence

A test run describes behavior under controlled conditions at a point in time. The NAIC supplement also asks how performance is monitored after deployment. An examiner may reasonably want to see what the system was permitted to do and what it did in production, in addition to what it did in testing.

Trussed does not perform bias testing and is not a pre-deployment testing or red-teaming product. It sits in a different place. Trussed enforces policy in the execution path, with actions such as allow, redact, block, rate limit and escalate for human approval, at the level of individual interactions and tool calls. Enforcement is at runtime, not reviewed after the fact, and it produces audit-ready evidence mapped to NAIC and NYDFS.

In practice, a test record can reference the policy in force for the tested system. Production evidence then shows how that policy applied to real interactions and tool calls. The two sets of records answer different questions and should be linked by system ID, not merged. Runtime evidence does not replace a test run, and a test run does not show production behavior.

Adoption decisions: depth, ownership and checks

NYDFS ties documentation depth to the complexity and materiality of the AI system. A tiered template follows from that. Require every field for systems that influence underwriting, pricing or claims outcomes. Allow a shorter set for lower-materiality uses, but keep the run ID, scope, result and sign-off for all of them.

The NAIC frames testing within a broader program with governance, risk controls and internal audit, so each record should link to program policies and name an accountable role. Set the retention period and the sign-off standard internally, justify them in writing and confirm them with counsel and any applicable state requirements.

Before rollout, check whether you can retrieve, for any AI system, every bias test run with its date, scope, data, metrics, thresholds, results, remediation and approver. Check that each run states its trigger. Check that vendor-supplied systems carry the testing evidence you rely on, recorded alongside your own runs. If a retrieval takes days of searching, the template is not yet the system of record.