Insurance AI Model Inventory Audit: A Step-by-Step Implementation Guide

An insurance AI model inventory audit verifies documentation completeness, ownership accountability, risk tier assignment, model lineage and change history, and the existence of ongoing monitoring controls, including for third-party and vendor-supplied models.

What the audit needs to cover

An AI model inventory audit for an insurance carrier examines whether every AI or machine learning model in active use, across underwriting, claims, pricing, and fraud detection, is captured in a documented inventory with clear ownership, risk classification, and lineage records. This scope reflects the direction set by the NAIC Model Bulletin on the Use of Artificial Intelligence Systems by Insurers, adopted December 4, 2023, which recommends insurers maintain a written AI Systems Program covering governance, risk management, and internal controls.

Colorado’s algorithm governance regulation and New York DFS Circular Letter No. 7 add jurisdiction-specific expectations around testing for unfair discrimination and documented oversight of external consumer data and information sources. Because state adoption of the NAIC Model Bulletin is not uniform, the audit’s first task is confirming which regulatory expectations apply based on the carrier’s licensed states, rather than assuming a single national standard.

Regulatory frameworks shaping the audit

Four reference points commonly shape how carriers scope inventory audits and related governance work:

• NAIC Model Bulletin (2023) Recommends a written AI Systems Program covering governance, risk management, and internal controls for insurer AI use.
• Colorado SB21-169 / Regulation 10-1-1 Requires governance and testing frameworks for external data and predictive models to prevent unfair discrimination, initially for life insurers.
• NY DFS Circular Letter No. 7 Directs insurers to maintain a documented AI governance framework with board and senior management oversight.
• NIST AI RMF 1.0 Referenced by NAIC guidance as a structuring tool organized around Govern, Map, Measure, and Manage functions.

Why insurance model inventories are hard to audit

Several structural issues make inventory audits harder in insurance than a simple asset count would suggest:

• Fragmented ownership: Models built by different business units for underwriting, claims, and pricing often lack a single accountable owner recorded in a central inventory.
• Vendor and third-party models: NAIC, Colorado, and NY DFS guidance each treat vendor-supplied AI systems as in-scope, but these are frequently excluded from internal inventories.
• Inconsistent risk classification: Without a shared framework, models are not consistently tiered by risk, making it difficult to prioritize testing for unfair discrimination.
• Jurisdictional variation: Terminology and requirements differ between NAIC’s AI Systems framing and Colorado and NY DFS references to external consumer data and information sources.

Model inventory audit checklist

Use the following checks to structure fieldwork and evidence collection. Adapt depth by risk tier and jurisdictional exposure.

• Confirm every AI/ML model in underwriting, claims, pricing, and fraud detection appears in a single inventory of record.
• Record a named accountable owner for each model, including models owned by vendors or business units outside the model risk team.
• Assign a risk tier using a shared classification method that supports prioritization of discrimination testing and oversight.
• Capture model purpose, decision context, and the consumer-facing processes the model influences.
• Document lineage: data sources, training or update history, version identifiers, and material change records.
• Verify monitoring and control evidence: performance review cadence, drift or override handling, and audit logging where applicable.
• Include external consumer data and information sources tied to model inputs, consistent with Colorado and NY DFS expectations.
• Map each in-scope model to applicable jurisdictional requirements based on licensed states, not a single assumed national standard.
• Confirm the written AI Systems Program (or equivalent governance document) reflects current inventory scope and controls.
• Prepare findings for senior management and board-level review, not only for retention inside the model risk function.

Sustaining compliance after the audit

An inventory audit identifies gaps at a point in time. Sustained compliance depends on governance processes that continue between formal review cycles.

• Report findings to senior governance bodies: The NAIC Model Bulletin’s emphasis on board and senior management oversight suggests audit results should be reviewed at that level, not just retained within a model risk team.
• Maintain the written AI Systems Program: Treat the AISP as a living document, updated as new models are deployed or existing ones are retired.
• Schedule recurring discrimination testing: For in-scope lines under Colorado’s framework, testing for unfair discrimination should be repeated on a defined cycle rather than performed once.
• Track ownership and change control continuously: Runtime monitoring, audit logging, and agent permission controls can support ongoing visibility into model and agent activity between formal audit cycles, which is relevant where insurers are extending AI use into agent-based workflows.

Frequently asked questions