NAIC AI Bulletin Compliance Tool Proof of Concept: Test Cases

What a runtime PoC can and cannot demonstrate

The NAIC Model Bulletin on the use of AI systems by insurers was adopted by the NAIC on December 4, 2023. Its text addresses the state insurance department with a placeholder, so it takes effect through each state's own issuance. As of October 2026, check the current state of issuance for each jurisdiction where you operate.

The bulletin sets out expectations for how insurers govern the development, acquisition and use of AI systems. It also lists documentation a department may request, including documentation of compliance with applicable AI Program policies, protocols and procedures. That last item is where a runtime tool can contribute. If every enforcement action is recorded and tied to a written policy, those records are one form of evidence that a policy was applied.

The limits matter as much as the fit. The bulletin materials encourage verification and testing methods to identify errors and bias in models, and those belong to pre-deployment and ongoing model testing. Third-party oversight also depends on contracts and audit processes. A runtime PoC does not establish any of these, and it does not establish drift detection unless the tool separately provides it. Treat the scenarios below as practitioner judgment about how to observe runtime behavior, not as NAIC requirements. Do not describe results as NAIC tool compliant. The NAIC's AI Systems Evaluation Tool is a regulator-facing instrument that has existed in several versions. Describe your results as evidence supporting your own AIS Program documentation.

Define the policy, the evidence record and the pass/fail rules first

Start with the written policy, not the tool. One stakeholder proposal for examinations describes evaluating policy, procedure and controls first, then sampling AI systems and documentation to validate adherence. A PoC can follow the same order: confirm the written policy, then sample enforcement records against it.

Before any scenario runs, agree on the evidence fields you will verify. The bulletin does not prescribe them, so these are your own criteria: policy version applied, action taken, timestamp, the application or actor involved, retention period, and export format. Also document any thresholds, rules or classifiers the tool uses, since the bulletin materials refer to techniques, measurements and thresholds as controls. Test them with known inputs.

Set pass/fail per scenario in three parts: the action matches the policy, a complete record exists, and the record can be exported and read without vendor involvement. A tool that blocks correctly but leaves no reviewable record fails the evidence half of the test.

Six runtime test cases

Each case pairs a scenario with an expected outcome and the bulletin theme it supports. Adapt the specifics to your own policy and use synthetic data.

• 1. Consumer personal information in a prompt (redact): Send a prompt containing synthetic policyholder identifiers to an internal model. Expect the identifiers to be redacted before reaching the model, with a record showing the policy version and action. Theme: risk management and internal controls. Include varied formats to measure redaction misses.
• 2. Claims or underwriting prompt needing review (escalate): Submit a prompt that your policy says requires human approval, such as one that would support an adverse consumer decision. Expect the interaction to pause, route to the named approver, and record the approver, the decision and the time. Theme: written AIS Program and accountability.
• 3. Out-of-scope tool call to an internal system (block): Have an agent attempt a call to a system or operation your policy does not permit. Expect a block, with the tool call recorded separately from the prompt that triggered it. Theme: internal controls.
• 4. Permitted interaction (allow): Run a routine, in-policy request. Expect it to pass and still generate a complete record. Allowed traffic is part of the sample an examiner would draw, so absent records for allows is a failure.
• 5. Third-party AI provider (policy and traceability): Route an interaction to an external AI provider and apply the same policy. Check that the evidence traces to the policy or contract term being enforced. Theme: third-party AI systems and data. This does not demonstrate contract audits.
• 6. Path bypass attempt (gap visibility): Try to reach a model or internal system without passing through the enforcement point. Expect the attempt to fail, or the gap to be visible and documented. Per-interaction evidence only exists where the tool is in the path.

Verify the evidence artifacts, not just the outcomes

For a sample from each scenario, pull the records and read them as an examiner would. Each record should show the policy version applied, the action taken, a timestamp, and which application or actor made the request. Prompts and tool calls should be distinguishable, since tool calls to internal systems carry different risk than text exchanged with a model.

Then test change control. Modify a policy, record who approved it, rerun a scenario, and confirm that new records show the new version while earlier records keep the old one. Escalation decisions should be tied to the same record trail. The sources place policy ownership and approval inside the AIS Program, so the tool should reflect your process rather than define it.

Finally, test retention and export. Set a retention period of your choosing, export a defined date range in a format your team can open, and have someone outside the PoC team find a specific interaction from a case reference. The bulletin materials do not specify retention periods or formats, so your criteria should come from your own records policy and counsel.

Interpreting results across jurisdictions and over time

Because the model bulletin takes effect through each state's issuance, carriers operating in several states should confirm the current requirements in each one and record any differences as additional acceptance criteria. Keep the core suite constant and add jurisdiction-specific scenarios rather than building separate PoCs.

Also expect the regulator-facing materials to keep changing. The NAIC evaluation tool and its supplement exist in multiple versions, and one comment mentions a possible future pilot. Review current versions when the PoC starts and again before you rely on the results. The supplement's stated purpose, obtaining information on the AIS Program, internal controls, third-party AI systems and data, and the process for acquiring and relying on them, is a useful checklist of what your wider documentation must cover.

A realistic outcome is a scorecard with passes, partial passes and documented gaps. Gaps outside the enforcement path, such as AI providers or applications that bypass the tool, should be listed with an owner and a plan, because they are the places where runtime evidence will not exist.

Where an in-the-path tool fits

Trussed enforces policy in the execution path across models, agents, tools and MCP servers, with outcomes of allow, redact, block, rate limit and escalate for human approval on individual interactions and tool calls. Enforcement generates audit-ready evidence that is mapped to NAIC and other regimes, and the tool can be deployed self-managed in the customer cloud with configurable retention and data residency. Trussed is built for runtime control and is not a pre-deployment testing product or a replacement for the AIS Program.

Whichever tool you evaluate, the six cases above apply unchanged. A vendor demonstration shows a tool working once. A defined suite shows whether its outcomes and records hold up against your own written policy.