Pilot acceptance criteria that survive procurement: writing the pass/fail before the cameras arrive

On a production line running 11,520 units per day across 8,000+ product variants, the question "did the pilot pass?" should have a binary answer. When it does not — when the vendor says it passed and the quality team says it did not — the dispute almost always traces to the same root cause: the pass/fail criteria were not written down before the cameras were installed.

This post is a practical template for setting vision pilot acceptance criteria before a pilot begins. It covers five components that evaluation teams routinely leave unspecified, the consequence of each omission, and the format for documenting criteria in a way that survives handoffs between the technical team and procurement.

Why acceptance criteria fail in practice

Vision pilots fail for two distinct reasons. The first is a system that cannot perform against your requirements. The second — and more common cause of disputed outcomes — is a system that performs against criteria nobody agreed on, or criteria that shifted between the pilot start and the go/no-go decision.

Common gaps that produce post-pilot disputes:

• No stated escape budget: the vendor demonstrates low FAR on their evaluation set, but "low" is interpreted differently by the quality team and the procurement team
• No FRR ceiling: the system catches defects but generates enough false alerts that rework cost exceeds inspection savings, and neither the contract nor the pilot success criteria addressed it
• No sample plan: the pilot ran against a cherry-picked part mix that did not represent the full production variance
• No model-freeze date: the vendor kept retraining throughout the pilot, and the system that ships to production does not match what was evaluated
• No exit clause: the vendor delivered a system that underperformed, and the buyer had no contractual mechanism to exit or recover costs

The solution to each of these is not negotiation after the pilot. It is specification before the pilot starts. Reading vision vendor accuracy claims: FRR, FAR, and escape rate covers the underlying performance metrics. This post focuses on translating those metrics into contractual acceptance criteria.

Component 1: Sample plan

The sample plan defines which parts will be inspected during the pilot and in what quantities. Without a sample plan, the vendor can train and evaluate against whatever set produces the strongest figures. With a sample plan, both sides know what the pilot is actually testing.

Minimum sample plan specification:

• Part variants in scope: Name specific part numbers or variant families. On a high-mix line running thousands of variants, specify whether the pilot covers a representative cross-section of the variance range or a defined subset of high-volume variants. A pilot scoped to three variants from an 8,000-variant line cannot be generalised to the full product estate.
• Defect classes in scope: List the specific defect classes the model will be evaluated against. Any defect class not listed is out of scope by default and cannot be counted as an escape or a false reject under the acceptance criteria.
• Sample size per defect class: Set a minimum number of confirmed defective parts per defect class. For statistically meaningful FRR and FAR figures, 30 confirmed defects per class is a practical floor. Below this count, the statistical variance makes figures nearly uninterpretable — a vendor can achieve apparent compliance by chance at low sample counts.
• Ground truth ownership: Specify who labels the ground truth classification for each sample. Vendor-labelled ground truth can inflate vendor-side scores at the margin cases where good and defective are genuinely ambiguous. Independent labelling or joint labelling with a documented boundary definition is more defensible.

Component 2: Escape budget (FAR ceiling)

The escape budget defines the maximum false accept rate the system must achieve to pass the pilot. Specify it as a rate — defects per million parts, or as an absolute count over the pilot volume — not as a percentage of the vendor's quoted accuracy.

When setting the FAR ceiling:

• Derive it from your FMEA and your quality agreement with your customer, not from what the vendor offers as an achievable benchmark
• Specify which defect classes the FAR ceiling applies to — critical and non-critical defect classes may warrant separate ceilings
• Define whether downstream inspection catches count as escapes: if your end-of-line sampling programme catches an escaped defect before shipment, that is still a system escape and should be counted as one
• Make the FAR ceiling binding: the pilot passes only if FAR at the agreed operating threshold is at or below the ceiling across the full pilot volume, not just a measurement window the vendor selects

A FAR ceiling stated without a defect class scope and a stated operating threshold is not enforceable. Both must be specified in the same document.

Component 3: FRR ceiling

The FRR ceiling defines the maximum false reject rate the system may generate. This is the acceptance criterion most consistently omitted, and the omission is what generates complaints about rework cost after deployment.

Set the FRR ceiling against your actual rework capacity. A line running 11,520 units per day with a rework bench that handles 200 re-inspections per shift can sustain a maximum FRR of roughly 1.7% at that throughput. An FRR ceiling set above your rework capacity is not a safeguard — it is a number that will be exceeded the first week the line runs at capacity.

Specify:

• Maximum acceptable FRR at the same operating threshold specified for the FAR ceiling. The FRR and FAR ceilings must both apply at the same threshold, not independently at different threshold settings.
• Whether FRR is measured across the full pilot volume or against a steady-state period following initial model tuning. If tuning is permitted during the pilot, the acceptance criteria should state clearly when the measurement period begins.
• What happens if the threshold that meets the FAR ceiling produces FRR above the FRR ceiling. This is the most common post-pilot technical dispute, and the contract needs a resolution clause that addresses it before the pilot starts.

Component 4: Timeline and milestone gates

A pilot without a defined end date is not a pilot — it is an open-ended evaluation. Specify:

• Pilot duration: A fixed calendar period from the first production run, typically 4–8 weeks for a single product family at production volume. The duration should allow enough volume to generate statistically reliable figures against the sample plan.
• Milestone gates: At least one interim review, typically at the midpoint, where FRR and FAR data are reviewed against targets. If the system is tracking to target, the pilot continues. If it is significantly off target, the parties either agree a documented recovery plan with a specific completion date or trigger the exit clause.
• Model-freeze date: A date after which the vendor may not retrain the model on new production data. Without a freeze date, the vendor can continue retraining through the evaluation window, and the FRR and FAR at the go/no-go decision will not predict the system's behaviour in ongoing production. Set the model-freeze date at the midpoint or earlier, depending on the vendor's typical characterisation timeline.
• Go/no-go decision date: A firm date by which both parties commit to a binary decision. Without a decision date, pilots extend indefinitely and consume engineering resources without a commercial outcome.

Component 5: Exit clause

An exit clause specifies the consequences when the pilot does not meet acceptance criteria. Agree it before the pilot starts, not after a failure, when both parties have invested in the outcome and the negotiating dynamic has shifted.

Exit clause components to specify:

• Trigger conditions: Which criteria failure invokes the exit clause — FAR above ceiling, FRR above ceiling, no decision at the milestone gate, or expiry without meeting targets. Specify whether any single criterion failure is sufficient to invoke the clause or whether multiple failures are required.
• Data ownership on exit: Who owns the ground-truth labels, defect annotations, and inference logs generated during the pilot? Labelled production data has independent value. If your team generated it, you should retain it regardless of outcome.
• Hardware return: What happens to cameras, lighting, edge compute hardware, or mounting structures installed during the pilot? Specify a removal timeline and condition requirements.
• Cost allocation: Who bears the cost of installation, removal, and any production downtime attributable to the pilot setup?

The exit clause is where procurement should be most closely involved. The technical team evaluates FRR and FAR against criteria. Procurement understands contractual exposure and cost allocation. Both need to have reviewed and signed the acceptance criteria document before the pilot begins.

Why procurement should co-sign the technical spec

Vision pilots typically start in the engineering or quality team and reach procurement only when a purchase order needs approval. By that point, the acceptance criteria have been defined informally in emails or verbal agreements, and procurement is reviewing commercial terms without visibility into what "passing" actually means.

The acceptance criteria document is a commercial document. The escape budget, the FRR ceiling, the model-freeze date, and the exit clause all have direct cost implications — rework costs, line downtime, hardware costs, and potential claim exposure if an escaped defect reaches a customer. Procurement should review and sign the acceptance criteria document at the same time as the commercial terms, not after the pilot has started.

A vendor agreement without a co-signed acceptance criteria document leaves the buyer's commercial exposure undefined. When a pilot result is disputed, the absence of documented criteria is not a neutral outcome — it generally favours the vendor, who will argue their system performed as expected against the absence of a specified target.

For the switching costs that compound when a pilot converts to a multi-year production deployment, and how exit terms set during the pilot affect your vendor options later, see From pilot to fleet: the real switching costs off proprietary vision. Acceptance criteria negotiated at pilot entry are much cheaper than exit negotiations two years into a production deployment.

Acceptance criteria template

Use this table as the starting framework. Fill in each field before the pilot agreement is signed. Both parties should date and sign a completed version.

HyperQ AI Vision pilots are structured around a written acceptance criteria document before cameras are installed. The pilot agreement specifies the escape definition, the FAR and FRR ceilings, and the operating threshold at which both are evaluated — so the go/no-go decision is a calculation, not a negotiation.

Frequently asked questions

How long should a vision pilot run?

Four to eight weeks is a reasonable range for a single product family at production volume. Shorter pilots do not accumulate enough parts to generate statistically reliable figures across the full defect class distribution. Pilots extending past eight weeks without a model-freeze date create a moving target — the system that ships to production may behave differently from what was evaluated at the go/no-go point. If the vendor needs more than eight weeks to characterise the system on your parts, ask why before the pilot agreement is signed.

Who should own the go/no-go decision?

The technical team should measure FRR and FAR against the acceptance criteria. The commercial decision — proceed, renegotiate, or exit — should require sign-off from both the technical lead and a procurement or commercial representative. Technical sign-off without commercial sign-off leaves the organisation commercially exposed. Commercial sign-off without technical sign-off produces a purchasing decision disconnected from actual system performance.

What if the vendor wants to retrain the model after the go/no-go decision?

Retraining after a go/no-go decision should be treated as a change-control event under your quality system, not as routine maintenance. The model that was evaluated is the model you agreed to deploy. Any subsequent retraining changes the system's behaviour and requires re-validation against acceptance criteria before production use. A model-freeze date in the pilot agreement prevents this from happening during the evaluation window; a change-control process handles it after production deployment begins.

Can we run the pilot on a smaller test set if our defect rate is very low?

Lower defect rates require larger sample volumes to accumulate enough confirmed defectives per class. At a 0.1% defect rate, accumulating 30 confirmed defectives per class requires 30,000 parts through the system — well above what most pilots budget. In practice, defective samples need to be supplemented with confirmed-defective parts from historical rejection bins or from controlled introductions. Supplement samples should be documented in the sample plan as such, since their defect presentation may differ from spontaneous production defects.

What if the vendor exceeds the FRR ceiling at the threshold that meets the FAR ceiling?

This is the most common technical dispute in pilot evaluations and the one most likely to be missing from standard pilot agreements. Resolution options: the vendor demonstrates an alternative threshold configuration that meets both ceilings simultaneously; the buyer accepts a revised FAR ceiling to enable a lower FRR; or the exit clause is invoked because the system cannot meet the dual constraint. All three outcomes need to be specified in advance. A pilot agreement that sets FAR and FRR ceilings without a resolution clause for simultaneous violation leaves both parties arguing past each other.

Share your current draft pilot agreement or acceptance criteria document and we will identify the gaps against this template, fill in the FRR and FAR ceilings appropriate to your line volume and rework capacity, and return a completed version within two weeks. No contract is required until both sides have agreed to the criteria in writing.

Share your pilot agreement draft and receive a reviewed acceptance criteria template with your specific metrics within two weeks, no contract until criteria are finalised.