VaulithNine questions, free
The canon

The twenty-five questions a carrier reviewer actually sends.

Drawn from the NAIC AI Systems Evaluation Tool, the NAIC AI model bulletin and the third-party provisions of 23 NYCRR 500, grouped the way the evaluation tool groups them. Each item shows the obligation it discharges, and each has its own page with the answers that make a reviewer stop on it. Vaulith scores against exactly these 25, and no other tool publishes the questions it scores against.

Nine questions, free. See what they stop on.The full assessment, $499 →

Governance

  1. AIS-1.1critical, weight 3

    Describe the AI Systems Program, including governance structure, named accountability, and the process by which AI systems are approved for production use.

    NAIC AI model bulletin §3, AIS Program
  2. AIS-1.2standard, weight 2

    Identify the individual accountable for AI governance and describe their reporting line to senior management.

    NAIC AI model bulletin §3.1
  3. AIS-1.3contextual, weight 1

    Describe how AI governance decisions, including approvals and exceptions, are recorded and retained.

    NAIC AI model bulletin §3.1

Inventory

  1. AIS-2.1critical, weight 3

    Provide a complete inventory of AI systems used in delivering the service, including the insurance function each supports.

    NAIC AI Systems Evaluation Tool, AI use inventory
  2. AIS-2.2critical, weight 3

    For each AI system, state whether it affects a consumer outcome, who made that determination, and on what basis.

    NAIC AI model bulletin §2, consumer-impacting AI
  3. AIS-2.3standard, weight 2

    Describe the risk classification applied to each AI system and the criteria that place a system in the highest tier.

    NAIC AI Systems Evaluation Tool, high-risk model identification
  4. AIS-2.4standard, weight 2

    Provide current model documentation for each high-risk system, reflecting the version now in production.

    NAIC AI Systems Evaluation Tool, model documentation

Data

  1. AIS-3.1standard, weight 2

    Describe the categories of data used as inputs to each AI system, including any consumer data.

    NAIC AI Systems Evaluation Tool, Exhibit D, data integrity
  2. AIS-3.4critical, weight 3

    For each high-risk model, describe the data used in training, including provenance, third-party sources, and any use of consumer data.

    NAIC AI Systems Evaluation Tool, training data provenance
  3. AIS-3.5standard, weight 2

    Describe the de-identification or minimisation applied to consumer data before it is used in training.

    NAIC AI model bulletin §4.2
  4. AIS-3.7critical, weight 3

    Describe independent validation performed on each high-risk model, including who performed it and when.

    NAIC AI model bulletin §4.3, validation and testing

Third parties

  1. AIS-4.1standard, weight 2

    Describe the due diligence performed on third parties whose models or data are incorporated into the service.

    NAIC AI model bulletin §5, third-party AI systems
  2. AIS-4.2critical, weight 3

    Identify all third parties whose AI systems, models, or model outputs are incorporated into the products or services provided to the insurer, including processing location.

    NAIC AI model bulletin §5.1 · 23 NYCRR 500.11(a)
  3. AIS-4.3standard, weight 2

    Describe contractual provisions governing the third party’s use of insurer data for model training.

    NAIC AI model bulletin §5.2

Testing

  1. AIS-5.1standard, weight 2

    Describe testing performed to detect unfair discrimination in AI system outputs, including methodology, protected classes considered, and cadence.

    NAIC AI model bulletin §4.4, unfair discrimination testing
  2. AIS-5.2standard, weight 2

    Describe ongoing performance monitoring and the thresholds that trigger intervention.

    NAIC AI Systems Evaluation Tool, drift and monitoring
  3. AIS-5.3contextual, weight 1

    Provide the most recent testing results for each consumer-impacting system.

    NAIC AI model bulletin §4.4

Oversight

  1. AIS-6.1standard, weight 2

    Describe where a human reviews or can override an AI system output before it reaches a consumer.

    NAIC AI model bulletin §4.5, human oversight
  2. AIS-6.3critical, weight 3

    Describe human review and override capability where an AI system contributes to an adverse consumer outcome, and provide override rates.

    NAIC AI model bulletin §4.5
  3. AIS-6.4contextual, weight 1

    Describe how consumers are informed that an AI system was used in a decision.

    NAIC AI model bulletin §4.6, consumer disclosure

Security

  1. SEC-1.1standard, weight 2

    Describe the information security controls applicable to the insurer’s nonpublic information held or processed by the third party.

    23 NYCRR 500.11(a)
  2. SEC-2.1standard, weight 2

    Describe access controls, including multi-factor authentication, applied to systems holding insurer nonpublic information.

    23 NYCRR 500.12
  3. SEC-3.1standard, weight 2

    Provide evidence of penetration testing performed within the last twelve months.

    23 NYCRR 500.5(a)
  4. SEC-4.1contextual, weight 1

    Describe the incident notification obligation and timeline owed to the insurer.

    23 NYCRR 500.11(a)(4)
  5. SEC-5.1contextual, weight 1

    Confirm the retention period applied to insurer nonpublic information after contract termination.

    23 NYCRR 500.13