VaulithNine questions, free
The canon · Testing · AIS-5.3

Provide the most recent testing results for each consumer-impacting system.

This is one of the twenty-five questions a carrier reviewer sends a vendor, in the words an examiner uses. Below it: the answers that make a reviewer stop on this item, derived by running Vaulith's rules engine on an estate that gets everything wrong, and the free questions that reach it.

What stops a reviewer here

2 findings reach this item.

Will escalateR-CONTRADICTION-BIAS · degrades this item

Your policy commits to scheduled fairness testing the estate does not show

AI policy (3.3) commits to testing fairness or disparate impact on a set schedule. MDL-001 Claims triage does not meet it, because no test is on record at all. A reviewer who reads the policy first expects a test result from the last ninety days to be in the file, and its absence tells them the schedule exists on paper. Once they have found one control that exists only on paper, they check the others the same way.

To fix: Run the test the policy already requires and record the date, or amend the policy to the cadence you actually keep. A commitment you do not meet costs more than one you never made.

Will escalateR-NO-MONITORING · degrades this item

No performance monitoring on record for MDL-001

MDL-001 Claims triage is not monitored in production against a threshold that would trigger a review, or you have not said that it is. The evaluation tool asks about drift because a model that was fair and accurate at validation can stop being either without anyone changing a line of code. For a system whose output reaches a consumer, a reviewer treats an unmonitored model as one whose current behaviour is unknown, and escalates rather than accepts it.

To fix: State the metric you watch, the threshold that triggers a review, and who receives the alert. A weekly report with one number and one named owner satisfies the question; a dashboard nobody is responsible for does not.

How it is scored

Credit per verdict, times the weight.

A contextual item carries weight 1. A pass earns 1.00 of it, a questioned answer 0.60, an escalated one 0.20 and a stop 0.00. An unanswered item earns nothing and still takes the citation penalty, so a profile cannot be improved by leaving an inconvenient question blank. The whole arithmetic is shown on the homepage and in every assessment.