The canon · Testing · AIS-5.2

Describe ongoing performance monitoring and the thresholds that trigger intervention.

This is one of the twenty-five questions a carrier reviewer sends a vendor, in the words an examiner uses. Below it: the answers that make a reviewer stop on this item, derived by running Vaulith's rules engine on an estate that gets everything wrong, and the free questions that reach it.

What stops a reviewer here

One finding reaches this item.

Will escalateR-NO-MONITORING · blocks this item

No performance monitoring on record for Claims triage (MDL-001)

Claims triage (MDL-001) is not monitored in production against a threshold that would trigger a review, or you have not said that it is. The NAIC bulletin lists model drift among the assessments an insurer keeps on its predictive models, because a model that was fair and accurate at validation can stop being either without anyone changing a line of code. For a system whose output reaches a consumer, a reviewer treats an unmonitored model as one whose current behaviour is unknown, and escalates rather than accepts it.

To fix: State the metric you watch, the threshold that triggers a review, and who receives the alert. A weekly report with one number and one named owner satisfies the question; a dashboard nobody is responsible for does not.

How it is scored

Credit per verdict, times the weight.

A standard item carries weight 2. A pass earns 1.00 of it, a questioned answer 0.60, an escalated one 0.20 and a stop 0.00. An unanswered item earns nothing and still takes the citation penalty, so a profile cannot be improved by leaving an inconvenient question blank. The whole arithmetic is shown on the homepage and in every assessment.