Across 47 production contracts, the most common source of post-deployment friction is not detection performance — it is the absence of a change-control procedure before the first model update is needed. A retrain happens, the model is promoted to production, and six months later the quality team cannot reconstruct which version was running during a batch under customer complaint. The model's detection rate is fine. The audit trail is not.
We have seen this resolved two ways. In the better case, the deployment team built the version-tracking record before the first retrain — a controlled document in the QMS with a row per model version, linked to the batch records it covered. In the other case, the version log lived in a shared drive folder, retrain dates were not recorded, and when a customer raised a complaint about a specific production week, the answer was "we believe it was version 3 but we cannot confirm." The customer's audit team did not accept "we believe."
Model change control is a quality-management discipline, not a data-science notebook convention. It belongs in the quality management system before the first update, not after the first audit finding.
Why model changes are different from software releases
Most manufacturing facilities have change-control procedures for machine parameters, tooling, software, and process recipes. Those procedures usually address: what changed, who authorised it, when it took effect, and how to roll it back. A vision model change sits inside that same framework but adds a dimension those procedures do not handle: the change affects judgment, not deterministic logic.
A recipe change to a CNC machine produces a predictable output shift. A model retrain does not — it improves performance on some inputs, degrades on others, and produces confidence-score distributions that shift in ways a simple before/after measurement does not capture. The standard change-control question "does the new version perform the same as the old version" is the wrong question for a model update. The correct question is "does the new version perform better on the cases we care about, and not worse on any case we care about" — and answering it requires a held-out validation set from the production environment.
This distinction is why model change control is more demanding than software change control, and why facilities that apply their existing software-change procedure to vision models without modification tend to under-document the validation evidence.
What counts as a change event
Not every model interaction requires formal change control. A useful taxonomy:
Requires full change-control procedure:
- Promotion of a new model version to production inference (replacing the active model)
- Addition of a new defect class to the taxonomy
- Expansion of the inspection zone or camera coverage
- Change of the acceptance threshold (confidence floor for pass/fail)
- Retraining on new images from a different process condition (new tooling, new raw-material lot, new surface treatment)
- Change of camera hardware, lens, or lighting at the inspection station
Requires documented record but not full procedure:
- Addition of images from the existing process distribution to augment the training set (same class, same condition, higher count)
- Threshold tuning within the range validated during initial deployment
- Configuration changes to pre-processing steps (crop region, image normalisation)
Typically not a model change event:
- Restarting the inference process after a server reboot with the same model version
- Logging configuration changes that do not affect inference
- UI or reporting configuration changes
The lines above are defaults; the quality manager is the authority on which category a specific change falls into for their context. Where the categorisation is uncertain, default to the full procedure.
Shadow-mode validation: the core of safe promotion
The correct method for validating a candidate model before production promotion is shadow mode: the candidate version runs in parallel with the production version, receiving the same camera feed, making its own decisions, and logging them — without those decisions affecting line output. Production continues under the current version while the candidate accumulates performance data from live parts.
The minimum shadow period is 24 hours on a continuously running line. For lines with strong shift-to-shift variation in part appearance, 72 hours captures the full visual distribution across shifts. For lines with known seasonal or material-lot variation, the shadow period should include at least one batch from each significant variation source.
At the end of the shadow period, compare candidate and production decisions on the same images. The comparison should cover:
- Agreement rate on clear-pass parts: the candidate should match the production version's pass decision on central-distribution good parts at near 100%
- Catch rate on confirmed defects: the candidate should equal or exceed the production version's detection rate on any held-out confirmed-defect set
- False-reject rate: the candidate's false-reject rate on confirmed good parts must not exceed the production version's by more than the defined allowance (typically ±0.5 percentage points)
- Boundary-case decisions: for any known boundary parts in the held-out set, the candidate and production versions should agree, or the disagreements should be reviewed by a quality engineer before promotion
If all four criteria are met, the candidate is eligible for promotion. If any criterion fails, the candidate is not promoted; the issue is investigated, additional training is applied, and the shadow period restarts.
This is more rigorous than most software change procedures. It is the appropriate level of rigour for a system that is making quality decisions on every unit the line produces.
Version numbering and what the record must contain
Each promoted model version should carry a version number, and the change record for each version should contain:
| Field | Content |
|---|---|
| Version number | Sequential, never reused |
| Promotion date and time | Exact timestamp |
| Trigger for change | Process change, new SKU, retraining request, drift detection |
| Summary of training data added | Date range, quantity, defect class, source production lot |
| Shadow-mode start and end | Dates and shift coverage |
| Shadow-mode outcomes | Agreement rate, catch rate, false-reject rate, any boundary-case notes |
| Acceptance criteria applied | The specific thresholds used for promotion decision |
| Authorised by | Name and role (typically quality manager or chief inspector) |
| Previous version archived | Version number and storage location |
| Rollback procedure on file | Yes/no, with reference to document |
The record does not need to be complex. A single row in a controlled spreadsheet or a form in the quality management system satisfies the requirement, provided the fields above are populated for every promotion event. The record must be retrievable by batch date — given a production batch under customer complaint, the quality team should be able to identify the exact model version that inspected it within minutes.
Rollback: designing for failure from the start
Before the first model is deployed, the rollback procedure should be documented and tested. The procedure should specify:
- The trigger condition for rollback (detection rate falls below threshold, false-reject spike exceeds limit, customer complaint traced to a specific batch)
- The person authorised to initiate rollback (typically the quality manager, not the IT team or the AI vendor's support desk)
- The time target from rollback decision to previous version reinstated (should be under 30 minutes for any line where inspection is rate-limiting)
- The post-rollback action: shadow-mode restart of the failed candidate for diagnosis, or retirement of the candidate with a new retrain from scratch
Testing the rollback procedure on a non-production environment before go-live is the equivalent of running a fire drill before the building opens. The first time a rollback is needed is not the time to discover that the previous version was overwritten during the promotion process.
The previous model version should be retained in storage for a minimum of 12 months, or for the duration of the batch retention period required under the quality standard that governs the line — whichever is longer. For medical-device supply chains under ISO 13485, this retention period may be 15 years or longer.
Mapping model change events to ISO 9001 and ISO 13485
| Change event | ISO 9001:2015 clause | ISO 13485:2016 clause | Required record |
|---|---|---|---|
| New model version promoted to production | 8.5.6 — Control of changes | 7.5.6 — Control of production changes | Change record with validation evidence |
| New defect class added | 8.5.6 + 8.6 — Release of products | 7.5.6 + 8.3 — Control of nonconforming product | Updated inspection specification + validation |
| Acceptance threshold changed | 8.5.6 | 7.5.6 | Record of parameter change and justification |
| Shadow-mode validation results | 8.6 — Release of products | 8.2.6 — Monitoring and measurement | Shadow-mode log with comparison outcomes |
| Rollback event | 8.5.6 + 10.2 — Nonconformity | 8.5 — Corrective action | Rollback record + root-cause investigation |
| Model version active at time of complaint | 8.7 — Nonconforming outputs | 8.3 — Control of nonconforming product | Version log traceable by batch date |
A note on ISO 13485: the standard requires that changes to production processes be validated before implementation. Shadow-mode validation as described in this post constitutes a defensible validation record under that requirement, provided the shadow-mode protocol and outcomes are documented in the format the standard specifies for validation records.
QA wants freeze; production wants iteration: how to satisfy both
The change-control friction in most deployments comes from two legitimate concerns in direct conflict.
Quality assurance wants the inspection system frozen: a stable, auditable baseline against which customer complaints, escapes, and regulatory submissions can be evaluated. Any change to the inspection system potentially affects the validity of prior quality records.
Production wants the model to improve: as new defect types are encountered, as process conditions shift, as new SKUs are introduced, the model should keep pace with the line, not lag behind it.
Shadow-mode validation is the mechanism that resolves this tension. It gives quality the validation evidence needed to treat a model update as a controlled change with a documented approval record, not an uncontrolled modification. It gives production the ability to push model improvements through on a cycle that matches engineering-change order cadence, not the slower rhythm of manual requalification.
The practical outcome is a change-control cadence rather than a freeze. A scheduled monthly review (examining detection-rate trends, false-reject trends, and any open engineering changes) produces a structured window for model updates. Changes are batched, validated in shadow mode, and promoted with a single authorisation record. The line is not frozen; it is managed.
Frequently asked questions
What is vision model change control?
Vision model change control is the set of documented procedures governing when a production inspection model can be updated, how updates are validated before they affect line output, what records are required for each update, and how to revert if a new version underperforms. It is the AI-specific application of production change management under ISO 9001 clause 8.5.6.
How often should a production vision model be retrained?
Retraining is event-driven, not scheduled on a fixed calendar. Triggers include: detection of drift in performance metrics (false-positive or false-negative rate trending outside control limits), the introduction of new SKUs or process changes, the accumulation of a significant number of confirmed boundary cases that the current model is handling inconsistently, or engineering change orders that alter part appearance within tolerance. Many deployments stabilise after an initial 60–90 day settling period and then retrain quarterly or less.
Can the AI vendor make model changes without quality-management approval?
No. In a properly structured deployment, the AI vendor (whether external or an internal data-science team) can prepare and validate a candidate model version. Promotion to production — the step that affects line output — requires authorisation by the quality function. The vendor operates the model; quality owns the production specification.
What is shadow mode in the context of AI vision?
Shadow mode is an inference configuration in which a candidate model version runs in parallel with the active production model, processing the same camera feed and generating its own decisions, but without those decisions driving the pass/fail output to the conveyor or the quality system. It allows the candidate's performance to be measured on live production data before it affects any output.
How does model version control integrate with an eQMS?
Each model version promotion event should generate a controlled document or record in the eQMS, linked to the relevant product lines and SKUs. The eQMS record should be retrievable by production batch date so that a query about which model was active on a given date can be answered directly. The vision data pipeline post in this series covers how inspection events, model version identifiers, and batch linkages flow into eQMS records from the inspection station.
What happens to the model change record during an ISO audit?
Auditors typically ask two questions: can you show me the validation evidence for the current model version, and can you show me which version was running during the batch under complaint. If the shadow-mode log and promotion record are in the QMS and linked to the batch record, both questions are answerable in minutes. If the change-control record is in a data scientist's notebook or an informal shared folder, neither question is answerable confidently.
For the data pipeline that carries model version identifiers and inspection events into eQMS batch records, see from frame grab to quality record: building the vision data pipeline into eQMS/MES. For the HyperQ AI Vision product overview, see the AI Vision solution page.
Send us your current quality management system structure — the standard you certify to, the change-control form you use, and the batch-record format for your highest-volume line. We will map the model change-control requirements to your existing procedure and return a gap analysis within 10 working days. No contract until the procedure fits your QMS.