A Tier-1 automotive parts supplier running 11,520 units per day across 8,000+ product variants asked three vision vendors for their inspection accuracy figures. All three came back above 99%. None of them measured the same thing — different test set compositions, different defect class scopes, different operating thresholds. The supplier was comparing three incompatible numbers without knowing it, and no question in the evaluation framework had asked the vendors to agree on a definition.
This situation is common enough to warrant a direct explanation: what standard accuracy metrics mean, why they are largely useless for manufacturing inspection, which numbers actually matter, and what to demand from a vendor before a pilot begins.
Why overall accuracy misleads in inspection
Accuracy measures the proportion of predictions a system gets right: (TP + TN) / (TP + TN + FP + FN). In most manufacturing contexts, this number is almost useless as an evaluation criterion. The reason is class imbalance.
On a line with a 0.3% defect rate, a system that classifies every single part as "pass" achieves 99.7% accuracy. It catches zero defects. The formula is not wrong — the metric is optimised for a problem that is not manufacturing inspection.
At 11,520 units per day, a 0.3% defect rate means roughly 35 defective parts per day on a single line. If the system never raises an alarm, those 35 parts proceed to the next operation or ship to a customer. A "99.7% accurate" system has done nothing. Vendors who quote overall accuracy understand this arithmetic. The ones who lead with it are expecting buyers who do not.
The class imbalance problem becomes more severe as defect rates drop. At 0.1% defect rate on the same line, a classifier that approves everything achieves 99.9% accuracy. Yet it passes roughly 11 defective parts per day. Single-number accuracy figures are a meaningful metric for balanced classification problems. Manufacturing inspection is not a balanced classification problem.
FRR and FAR: the numbers that reflect your actual costs
Manufacturing vision inspection has two failure modes with opposite economic consequences.
False reject rate (FRR) is the proportion of conforming parts the system incorrectly flags as defective. Every false reject requires manual re-inspection, adds handling time, and slows the line. On a high-volume line, FRR translates directly to rework headcount. When the false-alert rate rises high enough, operators begin overriding alarms to maintain throughput — and actual defects start passing as a consequence of lost trust in the system.
False accept rate (FAR) is the proportion of defective parts the system passes. These are your escapes. An escaped defect that reaches a customer carries warranty cost, recall risk, and potential field-failure consequences. In automotive-tier supply chains, a single confirmed escape can trigger a corrective action process whose total cost exceeds the annual cost of the inspection system.
FRR and FAR move in opposite directions as you adjust the detection threshold. Lower the threshold and you catch more defects (FAR falls) but reject more good parts (FRR rises). Raise the threshold and false alerts decrease (FRR falls) but escapes increase (FAR rises). No system eliminates both simultaneously. The shape of that tradeoff and whether it fits your quality and throughput requirements is the real evaluation question.
What an FRR@FAR operating curve tells you
An FRR@FAR curve — sometimes called an ROC or detection curve in this context — shows the system's false reject rate at each achievable level of false accept rate across the full threshold range. The curve is the product, not a single operating point. A vendor quoting one FRR and one FAR number is quoting their preferred point on this curve.
The question for your line is: at your required FAR, what does the FRR look like? If your quality plan allows one escape per 10,000 parts (FAR = 0.01%), what false-reject rate does that operating point require? If the answer is 8% FRR, a line running 11,520 units per day generates roughly 922 re-inspections per day on a single line — a staffing and floor-space decision, not just a quality metric.
Ask vendors to show you the full operating curve generated on your specific part samples and your specific defect classes. A vendor who cannot produce this curve either has not properly characterised their system or does not want you to see how the tradeoff looks on real production parts.
For a detailed explanation of how FRR is calculated from inspection data, see the false reject rate inspection formula guide on this site. The current post focuses on how to use FRR and FAR during vendor evaluation rather than on deriving them from first principles.
Confusion matrix requirements
A confusion matrix breaks down predictions into four cells: true positives (defects correctly flagged), true negatives (good parts correctly passed), false positives (good parts incorrectly flagged), and false negatives (defects incorrectly passed). For binary pass/fail inspection, this is a 2×2 table, and it is the minimum evidence a vendor should be able to produce.
Request the confusion matrix per defect class, not in aggregate. A system that performs well on surface scratches but misses inclusion defects will show strong aggregate figures if scratches dominate the test set. Per-class matrices show where the system has problems, which is what you need before committing to a production deployment.
Verify the test set composition before treating any confusion matrix as predictive. If the vendor validated on a 50% defect rate and your line runs at 0.5%, the FRR and FAR they report will not match your floor experience. Ask how many samples in the test set were defective, what the defect class distribution was, and what defect prevalence the test set used. A test set with far more defects than your actual production rate produces figures that look better than you will observe in production.
Escape rate: the definition varies by contract
"Escape rate" sounds standardised. In practice, vendors define it in at least three distinct ways:
- Defects-per-million parts passed by the vision system, regardless of downstream sampling catches
- Defects-per-batch reaching the customer, after accounting for downstream sampling
- Defects-per-part-type that the model was trained to detect, excluding defect classes outside the training scope
The third definition generates the most post-pilot disputes. A defect class your production introduced partway through the pilot — a new supplier's material, a changed process parameter, a new cosmetic standard — may fall outside the model's trained scope. Whether that counts as an escape in your contract depends entirely on what was written down before the pilot started.
Before signing any pilot agreement, define escape in writing: which defect classes are in scope, which counting unit applies, and whether downstream inspection catches count as escapes or not. A vendor who resists committing to a definition before the pilot starts is managing their own exposure, not yours.
Vendor evaluation decision table
The following table covers the questions to put to any vision vendor at each stage of evaluation. Use it as a pre-pilot screening instrument.
| Stage | Question | What the answer reveals |
|---|---|---|
| Initial demo | What defect classes were in the demo sample set? | Whether the demo reflects your parts or a curated set |
| Initial demo | What was the defect prevalence in the demo set? | High defect prevalence inflates accuracy figures |
| Proposal | Can you show an FRR@FAR curve for my specific part type? | Whether the vendor has characterised the performance tradeoff on production parts |
| Proposal | At my target FAR, what is the FRR? | Translates claimed performance into rework cost for your specific volume |
| Proposal | How is escape rate defined in your proposal? | Eliminates definitional mismatches before contract |
| Pilot agreement | How many confirmed defects are in the test set, per class? | Validates statistical validity of quoted FRR and FAR |
| Pilot agreement | Who labels the ground truth — vendor, buyer, or jointly? | Vendor-labelled ground truth can inflate vendor-side scores where ground truth is ambiguous |
| Mid-pilot | Can you show the per-class confusion matrix so far? | Reveals underperforming defect classes hidden in aggregate metrics |
| Mid-pilot | What has the FRR been over the pilot volume to date? | Measures operator burden, not just escape count |
| Go/no-go | Does the escape count match the agreed contract definition? | Verifies the definition held through the full pilot |
| Go/no-go | What was the aggregate FRR across the full pilot volume? | Final measure of rework cost before a production commitment |
Running a signal test before the full pilot
Before committing to a multi-week pilot, a brief signal test can separate vendors who have already solved your problem from those who will be characterising their system on your production time. Prepare 200 parts: 180 confirmed good, 20 confirmed defective across the defect classes you care about most. Do not disclose the composition. Request the confusion matrix per defect class, the FRR and FAR at the vendor's recommended operating threshold, and the FRR@FAR operating curve across the threshold range.
A vendor who completes this in a single evaluation session and can explain the tradeoff clearly is worth a full pilot commitment. A vendor who wants more parts, a different set, or more time before committing to any numbers is signalling that the characterisation work is not yet complete.
HyperQ AI Vision generates per-class confusion matrices and FRR@FAR curves as part of the pre-pilot evaluation process — on customer-supplied parts and customer-labelled ground truth, not internal benchmarks. For the contractual acceptance criteria that should accompany a full pilot, including how to set FRR ceilings and escape budgets, see Pilot acceptance criteria that survive procurement.
Frequently asked questions
What is an appropriate false accept rate target for industrial inspection?
It depends on the cost of an escape in your application. For automotive-tier components with field-failure or safety consequences, quality plans commonly require FAR below 0.1%. For cosmetic defects in consumer packaging where the consequence is a customer complaint rather than a field failure, 0.5–1% may fall within acceptable limits. The right target comes from your FMEA, your customer's quality requirements, and your quality agreement — not from a vendor benchmark.
Is FRR or FAR more important?
Both matter, but to different functions. Quality managers focus on FAR because escapes reach the customer and generate warranty or recall costs. Production managers focus on FRR because false rejects generate rework, handling time, and alert fatigue that degrades operator compliance. When these functions disagree on priorities, set a ceiling on both: a maximum FAR and a maximum FRR that the vendor must operate within simultaneously at the agreed threshold. A vendor who cannot meet both ceilings has not yet solved your problem.
Can a vendor guarantee zero escapes?
No credible vendor will guarantee zero escapes for a production vision system. What they can commit to is a documented FAR at a stated operating threshold on your specific parts and defect classes. Zero-escape claims in production inspection either apply to a narrow subset of defect classes, exclude failure modes most likely to reach a customer, or are measured on test conditions that do not match production.
What is the minimum test set size for reliable FRR and FAR figures?
As a working floor: 30 or more confirmed defective samples per defect class for statistically stable FAR estimates, and 1,000 or more confirmed good parts for FRR estimates at low false-reject rates. A validation on 50 parts with 10 defects produces numbers with confidence intervals wide enough to make the results nearly uninterpretable. Vendors proposing to validate on fewer than these volumes should be asked to explain how they plan to control for statistical variance before you commit to a pilot.
What does "operating threshold" mean in practice?
The threshold is the sensitivity setting that governs the pass/fail classification for each part. Lowering it makes the system flag more parts as potentially defective — catching more genuine defects but also raising more false alarms. Raising it reduces false alarms but lets more defects pass. In production deployment, the threshold is set to operate within your FRR and FAR ceilings simultaneously. It is a configurable parameter, not a fixed property of the model. When a vendor quotes performance figures, always ask: at what threshold, and how far can the threshold shift before the system breaches your quality plan limits?
Why do vendors quote accuracy instead of FRR and FAR?
Accuracy is a single number that presents well in a proposal. FRR and FAR require a two-dimensional explanation and force an honest conversation about the performance tradeoff. A vendor quoting only accuracy on a manufacturing inspection problem is presenting the most favourable framing their metrics allow. A vendor willing to show you the FRR@FAR curve on your own parts has already done the work to understand how their system behaves on your problem. These two categories behave very differently in production.
Send us a labelled set of 200 parts — 180 confirmed good and 20 from your highest-priority defect classes — and we will return a per-class confusion matrix, an FRR@FAR operating curve, and a recommended threshold within two weeks. No contract is required until the figures meet your quality plan targets.