Every AI vision vendor in your RFP claims 99.9% accuracy. Not one of them is lying. And not one of them is telling you the full story.
99.9% is a detection rate. It tells you what percentage of defects the system caught—on a specific dataset, under specific conditions, for a specific defect type. Change any of those three variables and the number changes with them. A system validated on vendor test data has not been validated on your product. 99.9% on their data and 99.9% on your data are not the same measurement.
Most vendors do not volunteer this distinction. That is worth noting before you sign anything.
The metric that actually costs you money
Detection rate gets quoted because it sounds good in a procurement deck. The metric that affects your production economics is the false reject rate—how often the system flags good product as defective.
A system with a 99.9% detection rate and a 2% false reject rate will stop your line, trigger re-inspection, create rework queues, and frustrate operators every single shift. The detection rate looks excellent in a quarterly report. The FRR shows up in your OEE, your line stoppages, and eventually in operator behaviour.
The vendor optimises for detection. The buyer pays for rejects.
FRR formula:
FRR = (False rejects / Total good parts inspected) x 100
Benchmark: Below 0.5% FRR while maintaining 99% defect detection. This is achievable—but only if you know what is driving your rejects.
The trade-off is real. In medical device manufacturing, practitioners validate to 99.99% confidence that all bad parts are caught—and explicitly accept that a few good ones will be sacrificed for that goal. The acceptable FRR depends on what you are making and what a defect escape costs downstream. For cosmetic consumer goods, a higher FRR is intolerable because the cost of a false reject exceeds the cost of a defect reaching the customer. For safety-critical components, you tolerate higher FRR because the cost of a defect escape is catastrophic.
Zero false rejects is a warning, not a win
An AI vision system with no false rejects in the past week is not necessarily a high-performance system. It may be a system tuned so permissively it is passing real defects. Or it may be a system that operators have taken offline.
The operator bypass problem. When FRR is high, operators lose trust. They press the bypass button. They run production for days with inspection disabled. A camera that can be switched off without consequences is not part of the system at all—it is an expensive gadget doing nothing. The 99.9% accuracy figure in the procurement deck becomes irrelevant the moment the camera is bypassed.
This is not hypothetical. Integration professionals report it as endemic—almost every inspection system has a bypass button on the HMI, and management often requires one. High FRR is not just a throughput problem. It is a system abandonment problem.
The proof-of-life insight. A quality engineer at a major brewery noticed that the vision system had stopped rejecting anything. No false positives. No rejects at all. He investigated rather than assumed success. The camera had failed. Six tractor trailers of product were already on the road.
He knew something most dashboards do not show: a non-zero false reject rate is the normal operating signal of a functioning system. When the signal disappeared, the correct response was not satisfaction—it was investigation.
False rejects are proof of life. A system below 0.5% FRR is optimised. A system at 0% FRR should be checked.
Four methods to reduce FRR without increasing false passes
1. Fix the lighting first. Inconsistent lighting is the single most common cause of high FRR. Shadows, reflections, and variation between shifts create image noise the model reads as anomalies. Standardise lighting before adjusting model parameters. This alone can reduce FRR significantly without touching the model. Lighting is 80% of the job.
2. Augment training data with diverse good-part images. A model trained on a narrow range of acceptable surface variation will over-flag parts that fall within spec but look slightly different from the training set. Add representative images of acceptable variation—different material lots, tooling wear states, environmental conditions. 1,000 images with proper diversity reduces FRR by training the model to distinguish real defects from normal production variation.
3. Calibrate thresholds by defect category. Not all defect types carry the same risk. A cosmetic scratch on a non-visible surface is not a dimensional error on a structural component. Set confidence threshold bands by defect category rather than applying a single threshold across all classifications. Critical defects get aggressive thresholds (accept higher FRR). Cosmetic defects get permissive thresholds (preserve throughput).
4. Run structured retraining cycles. A model deployed six months ago works with data from six months ago. Materials shift between suppliers. Tooling wears. Humidity changes with the season. The model may be rejecting parts it would have passed at deployment because production has drifted while the model stayed static. Quarterly retraining cycles keep the model aligned with what your line actually produces today.
Teams that address all four areas typically see 60 to 80 percent false positive reduction compared to rule-based vision systems. FRR below 0.5% becomes consistent, not occasional.
Before procurement: four metrics, three questions
Get all four metrics before any system goes into production:
- Detection rate. What percentage of actual defects does the system flag? Specify defect type and severity threshold.
- False positive rate. Of all alerts generated, what proportion are genuine defects? High false positive rate equals low operator trust.
- False reject rate (FRR). What percentage of good product gets pulled for re-inspection? This is the metric that hits throughput.
- OEE impact. Net effect on overall equipment effectiveness once false rejects, downtime from alerts, and re-inspection time are factored in.
Three questions for the vendor meeting:
- What dataset was this accuracy measured on?
- What is your false reject rate on production-representative data?
- How does the model maintain accuracy over time?
If a vendor cannot give you all four metrics—separately, from production-representative data—the 99.9% figure tells you very little.
The problem is not the metric. The problem is deploying it without the context that makes it meaningful. Talk to us about what your numbers actually look like.