Insurer AI Vendor Due Diligence Checklist: Third-Party Model Oversight
A practitioner checklist for evaluating third-party AI and machine learning vendors: what to require before signing, and what to monitor once the model is live.
Insurers should evaluate third-party AI/ML vendors against two distinct sets of requirements: pre-contract documentation (model validation, data lineage, bias testing) sufficient to satisfy NAIC and state accountability expectations, and post-deployment runtime oversight (monitoring, access logging, revalidation), since regulatory frameworks treat insurers as accountable for vendor model outcomes regardless of who built the model.
Vendor Due Diligence Workflow
Treat vendor review as a repeatable process that spans procurement and production, not a one-time approval step at signing.
- 1
Intake and Scoping
Identify the AI/ML use case, the decisions it will influence, and the data flows between the insurer and the vendor.
- 2
Documentation Request
Request model validation reports, data lineage records, and bias testing results before contract execution.
- 3
Independent Review
Assess vendor documentation against NAIC Model Bulletin expectations and applicable state DOI guidance rather than accepting vendor summaries at face value.
- 4
Contract Terms
Negotiate audit rights, a defined revalidation cadence, and termination provisions tied to ongoing model performance.
- 5
Deployment Monitoring Setup
Establish output monitoring, access logging, and escalation paths before the model goes live in production.
- 6
Periodic Revalidation
Re-run bias and performance testing on a defined schedule, and whenever the vendor materially updates the model.
Pre-Contract Documentation Checklist
Before procurement is finalized, request and review the following from any AI/ML vendor.
- Model validation reports covering methodology, training data sources, and performance benchmarks
- Data lineage documentation showing origin, transformations, and permissible use of training and inference data
- Bias and disparate impact testing results across protected classes and reasonable proxy variables
- Explainability documentation describing how outputs can be interpreted for adverse action and appeals handling
- The vendor's own model risk management policy and version control practices
- Evidence of the vendor's compliance mapping to the NAIC Model Bulletin and applicable state DOI guidance
- Data security and privacy attestations covering handling, retention, and breach notification
- Contractual audit rights and defined provisions for updated documentation as regulatory expectations evolve
Where Vendor Attestation Suffices vs. Where Technical Controls Are Required
Not every requirement needs the same level of scrutiny. Some items can rely on vendor representations with periodic audit rights; others require the insurer to run its own technical controls independent of vendor claims.
Data Source Documentation
Vendor-provided description of training data origin and permissible use, confirmed through periodic audit rights.
Development Methodology
Vendor summary of model architecture and development process, reviewed against internal model risk standards.
Security and Privacy Practices
Vendor attestations on data handling, retention, and breach notification, refreshed on a defined cycle.
Independent Bias Testing
Bias and disparate impact testing run on the insurer's own data, not solely relied upon from vendor-supplied results.
Output Monitoring
Real-time monitoring of model outputs in production to detect drift or performance degradation.
Access Logging
Logging of who queried the vendor model, when, and for what decision, retained for regulator examination.
Why Vendor AI Oversight Is a Regulatory Requirement
Regulators, through the NAIC Model Bulletin and corresponding state DOI adoptions, hold insurers accountable for the outcomes of AI and machine learning systems used in underwriting, claims, and pricing decisions, regardless of whether the model was built in-house or licensed from a third-party vendor. Delegating model development to a vendor does not delegate regulatory responsibility.
This means an insurer cannot treat a vendor's assurance that "the model is compliant" as a substitute for its own documentation review. Examiners expect the insurer to demonstrate governance over vendor models with the same rigor applied to internally developed ones, including evidence of data lineage, bias testing, and validation appropriate to the use case.
Regulatory frameworks treat insurers as accountable for vendor model outcomes regardless of who built the model.
Post-Deployment Runtime Oversight
Documentation review at the point of signing addresses only half of the compliance picture. Vendor models are frequently updated, retrained, or reweighted after deployment, often without a formal notification process to the insurer. Runtime oversight is what allows an insurer to detect when a model's behavior has drifted from what was originally validated.
Practical runtime oversight includes monitoring model outputs for unexpected shifts, maintaining access logs that show which internal users and systems queried the vendor model, and running independent revalidation on a defined schedule rather than relying solely on the vendor's own retesting. These controls give the insurer the evidence needed to respond to a regulatory inquiry about a specific decision or outcome.
Operationalizing the Checklist
A due diligence checklist is only useful if it is applied consistently across every vendor relationship, not just the largest or most visible ones. Operationalizing it typically means assigning clear ownership across compliance, legal, and actuarial or underwriting teams, so that documentation review, contract negotiation, and ongoing monitoring do not fall through organizational gaps.
Because vendor models change over time, the checklist should be treated as a living process rather than a one-time gate at procurement. Revisiting documentation and re-running validation on a regular cadence, and whenever the vendor materially updates the model, keeps the insurer's evidence current with what regulators expect to see during an examination.
Two Phases of Vendor AI Oversight
Vendor AI oversight breaks into two distinct phases, each with its own evidence requirements.
Pre-Contract Due Diligence
Documentation and testing required before procurement is finalized, covering validation, data lineage, and bias results.
Post-Deployment Oversight
Monitoring, access governance, and revalidation once the model is in production, addressing changes that occur after signing.
Non-Delegable Accountability
Insurers remain responsible for outcomes regardless of vendor ownership of the underlying model.
Operational Considerations After Contract Signing
Once a vendor model is in production, these operational practices keep oversight active rather than a one-time exercise.
- Configure real-time monitoring for output drift and performance degradation
- Maintain access logs showing who queried the vendor model, and when
- Schedule periodic bias and performance revalidation independent of vendor-supplied results
- Define escalation procedures for when vendor model outputs trigger adverse action review
- Track vendor model version changes and require revalidation on material updates
- Retain documentation demonstrating ongoing oversight for regulator examination
Structure Vendor AI Oversight Before and After Deployment
Insurers evaluating third-party AI vendors need both pre-contract documentation review and runtime visibility into how vendor models behave in production.
Talk to an Expert