VaulithNine questions, free
Resources · the canon

The twenty-five questions a carrier's AI review turns on.

Drawn by the Vaulith team from the NAIC AI model bulletin, the NAIC AI Risk Evaluation Supplement and the third-party provisions of 23 NYCRR 500, grouped by topic. No carrier sends this list; each carrier sends its own questionnaire, and these are the questions those questionnaires turn on. Each item shows the obligation it discharges, and each has its own page with the answers that make a reviewer stop on it. Vaulith scores against exactly these 25, and publishes every one with the paragraph it comes from.

Nine questions, free. See what they stop on.The nine questions, explained →What a reviewer sees →

Governance

  1. AIS-1.1critical, weight 3

    Describe the AI Systems Program, including governance structure, named accountability, and the process by which AI systems are approved for production use.

    NAIC AI model bulletin, Section 3 ¶1.2 and ¶3.1 · AI Risk Evaluation Supplement v5.0, Exhibit B
  2. AIS-1.2standard, weight 2

    Identify the individual accountable for AI governance and describe their reporting line to senior management.

    NAIC AI model bulletin, Section 3 ¶1.3 and ¶2.3 · AI Risk Evaluation Supplement v5.0, Exhibit B 2a
  3. AIS-1.3contextual, weight 1

    Describe how AI governance decisions, including approvals and exceptions, are recorded and retained.

    NAIC AI model bulletin, Section 3 ¶2.2

Inventory

  1. AIS-2.1critical, weight 3

    Provide a complete inventory of AI systems used in delivering the service, including the insurance function each supports.

    NAIC AI model bulletin, Section 3 ¶3.3(a) · AI Risk Evaluation Supplement v5.0, Exhibit A
  2. AIS-2.2critical, weight 3

    For each AI system, state whether it affects a consumer outcome, who made that determination, and on what basis.

    NAIC AI model bulletin, Section 2 (Adverse Consumer Outcome) and Section 3 ¶1.1 · AI Risk Evaluation Supplement v5.0, Exhibit B 3c
  3. AIS-2.3standard, weight 2

    Describe the risk classification applied to each AI system and the criteria that place a system in the highest tier.

    NAIC AI model bulletin, Section 3 ¶1.4 · AI Risk Evaluation Supplement v5.0, Exhibit B 3k
  4. AIS-2.4standard, weight 2

    Provide current model documentation for each high-risk system, reflecting the version now in production.

    NAIC AI model bulletin, Section 3 ¶3.3(b) and ¶3.7 · AI Risk Evaluation Supplement v5.0, Exhibit C

Data

  1. AIS-3.1standard, weight 2

    Describe the categories of data used as inputs to each AI system, including any consumer data.

    NAIC AI model bulletin, Section 3 ¶3.2 (data lineage, quality, integrity) · AI Risk Evaluation Supplement v5.0, Exhibit D
  2. AIS-3.4critical, weight 3

    For each high-risk model, describe the data used in training, including provenance, third-party sources, and any use of consumer data.

    NAIC AI model bulletin, Section 3 ¶3.2 (lineage) and ¶3.4 (training data suitability) · AI Risk Evaluation Supplement v5.0, Exhibit D
  3. AIS-3.5standard, weight 2

    Describe the de-identification or minimisation applied to consumer data before it is used in training.

    NAIC AI model bulletin, Section 3 ¶3.5 (protection of consumer information) · Section 3 ¶3.2 (data practices) · AI Risk Evaluation Supplement v5.0, Exhibit B 3d
  4. AIS-3.7critical, weight 3

    Describe independent validation performed on each high-risk model, including who performed it and when.

    NAIC AI model bulletin, Section 3 ¶3.4 (validation and testing) · Section 3 ¶2.3 (independence of lines of defense) · AI Risk Evaluation Supplement v5.0, Exhibit C

Third parties

  1. AIS-4.1standard, weight 2

    Describe the due diligence performed on third parties whose models or data are incorporated into the service.

    NAIC AI model bulletin, Section 3 ¶4.1 and Section 4 ¶2.1 · AI Risk Evaluation Supplement v5.0, Exhibit B 3l
  2. AIS-4.2critical, weight 3

    Identify all third parties whose AI systems, models, or model outputs are incorporated into the products or services provided to the insurer, including processing location.

    NAIC AI model bulletin, Section 4 ¶2.1 · 23 NYCRR 500.11(a)(1) · AI Risk Evaluation Supplement v5.0, Exhibit C
  3. AIS-4.3standard, weight 2

    Describe contractual provisions governing the third party’s use of insurer data for model training.

    NAIC AI model bulletin, Section 3 ¶4.2 and Section 4 ¶2.2 · AI Risk Evaluation Supplement v5.0, Exhibit B 3p

Testing

  1. AIS-5.1standard, weight 2

    Describe testing performed to detect unfair discrimination in AI system outputs, including methodology, protected classes considered, and cadence.

    NAIC AI model bulletin, Section 3 ¶1.1 and ¶3.2 (bias analysis) · AI Risk Evaluation Supplement v5.0, Exhibit C
  2. AIS-5.2standard, weight 2

    Describe ongoing performance monitoring and the thresholds that trigger intervention.

    NAIC AI model bulletin, Section 3 ¶3.3(c) (model drift) and Section 4 ¶2.4 · AI Risk Evaluation Supplement v5.0, Exhibit C
  3. AIS-5.3contextual, weight 1

    Provide the most recent testing results for each consumer-impacting system.

    NAIC AI model bulletin, Section 4 ¶2.4 · AI Risk Evaluation Supplement v5.0, Exhibit C

Oversight

  1. AIS-6.1standard, weight 2

    Describe where a human reviews or can override an AI system output before it reaches a consumer.

    NAIC AI model bulletin, Section 3 ¶2.3 and ¶3.1 · AI Risk Evaluation Supplement v5.0, Exhibit B
  2. AIS-6.3critical, weight 3

    Describe human review and override capability where an AI system contributes to an adverse consumer outcome, and provide override rates.

    NAIC AI model bulletin, Section 3 ¶3.1 and Section 2 (Adverse Consumer Outcome) · AI Risk Evaluation Supplement v5.0, Exhibit B 3c
  3. AIS-6.4contextual, weight 1

    Describe how consumers are informed that an AI system was used in a decision.

    NAIC AI model bulletin, Section 3 ¶1.9 · AI Risk Evaluation Supplement v5.0, Exhibit B 3n

Security

  1. SEC-1.1standard, weight 2

    Describe the information security controls applicable to the insurer’s nonpublic information held or processed by the third party.

    23 NYCRR 500.11(a)
  2. SEC-2.1standard, weight 2

    Describe access controls, including multi-factor authentication, applied to systems holding insurer nonpublic information.

    23 NYCRR 500.12
  3. SEC-3.1standard, weight 2

    Provide evidence of penetration testing performed within the last twelve months.

    23 NYCRR 500.5(a)
  4. SEC-4.1contextual, weight 1

    Describe the incident notification obligation and timeline owed to the insurer.

    23 NYCRR 500.11(b)(3)
  5. SEC-5.1contextual, weight 1

    Confirm the retention period applied to insurer nonpublic information after contract termination.

    23 NYCRR 500.13(b)