Skip to content

Trustworthy inspection

Beyond accuracy: defining an AI inspection system that can go live

A 99% aggregate score can hide one critical miss. Acceptance must return to risk, false alarms, misses and review cost.

About 1 min read

Back to the journal

Industrial anomalies include surface damage and logical errors in count, position and relationships. One aggregate number cannot evaluate them all.

Evaluate by risk, not one score

Aggregate accuracy is easy to misuse when normal samples dominate. Acceptance should stratify objects, inspection items and risk, then examine miss rate, false-positive rate, localization quality and review volume. A critical safety item needs a different threshold and human process from a cosmetic surface item.

Cover logical anomalies and allow abstention

MVTec LOCO extends anomaly detection beyond dents and scratches to logical constraints such as quantity and relative position. Selective-classification research also shows how a model can refuse uncertain predictions. In inspection, refusal is a designed safety boundary: automate clear work and route uncertainty or high risk to accountable people.

Make release criteria and results traceable

A production pilot should define covered defects, allowed review volume, recapture rules, human turnaround and version comparison before launch. Every metric must trace back to source images, inspection items, versions and confirmed outcomes. Only then does accuracy become operational quality information.

Academic sources

  1. Beyond Dents and Scratches: Logical Constraints in Unsupervised Anomaly Detection and LocalizationCVPR 2022

    Extends industrial anomalies to quantity, position and logical constraints.

  2. Selective Classification for Deep Neural NetworksNeurIPS 2017 Workshop / arXiv

    Studies abstention on uncertain inputs to trade coverage for lower prediction risk.