If your team is shortlisting AI vision vendors for a manufacturing line, you've probably moved past the demo stage. Now comes the harder part: separating genuine capability from polished marketing. Whether you're comparing established rule-based vision vendors, emerging AI specialists, or AI-native platforms like Hypernology's HyperQ AI Vision, the evaluation criteria should be the same--rigorous, production-grounded, and led by your engineering team.
Ten questions every manufacturing engineer and IT/automation lead should ask before signing anything.
1. Is the software hardware-agnostic, or does it require proprietary cameras?
Why it matters: Hardware vendor lock-in inflates cost and kills flexibility when you need to expand to new lines or replace aging equipment.
- Weak answer: "Our system works best with our own camera hardware"--this signals a bundled sale, not a software-first solution.
- Strong answer: Universal camera compatibility--support for third-party industrial cameras (Basler, FLIR, Sony) via GenICam/GigE Vision standards, with no performance penalty and 30--50% hardware cost savings by reusing existing infrastructure.
HyperQ AI Vision, for example, runs on customer-owned camera infrastructure, which matters when you already have line hardware in place.
2. How many training images are required per SKU, and what about rare defects?
Why it matters: In real production environments, defects like micro-cracks or surface voids may appear only a handful of times per quarter. A vendor that requires 500+ labeled defect images per class is impractical for low-frequency failure modes.
- Weak answer: "We recommend at least 300 images per defect type"--effectively unusable for rare anomalies.
- Strong answer: Few-shot or anomaly detection capability, where the model learns from as few as 10--30 images, or from normal-sample-only training for unsupervised defect detection.
Ask specifically: Does your system support unsupervised or semi-supervised training for rare defect types?
3. Can it manage 1,000+ product variants without retraining from scratch?
Why it matters: High-mix manufacturing environments--automotive components, electronics, consumer goods--routinely run hundreds of SKUs. Retraining for every new variant is operationally unsustainable.
- Weak answer: "You'll need to set up a new model for each product line."
- Strong answer: A shared model architecture that handles 8,000+ product variants with product-specific parameter sets, or few-shot adaptation that adds new SKUs using a small labeled batch without rebuilding the base model. Zero-configuration changeover--the system loads the correct inspection model in under 2 seconds from a PLC signal, with no operator input.
4. Does it support PLC handshake and existing line controllers?
Why it matters: AI vision doesn't operate in isolation. It needs to send pass/fail signals to PLCs (Siemens, Allen-Bradley, Mitsubishi) and connect with SCADA or MES layers.
- Weak answer: "We have an API you can build on top of."
- Strong answer: Native support for PROFINET, EtherNet/IP, or OPC-UA; pre-built connectors for common PLC brands; and documented integration timelines from past deployments.
Ask for a reference deployment that matches your line controller brand.
5. What is the realistic deployment timeline, and who does the integration work?
Why it matters: Vendors routinely quote best-case timelines. What matters is who owns the integration--and what happens when it overruns.
- Weak answer: "Deployment takes 2--4 weeks" with no clarification of scope, assumptions, or who provides on-site support.
- Strong answer: A phased timeline broken into camera calibration, model training, PLC integration, and production validation--with clear ownership at each stage and named local resources. On-site physical setup: 2 days. Full implementation from contract to production-ready: 4--8 weeks.
Days-to-deploy claims mean nothing without a defined scope. Push for a statement of work, not a slide.
6. How is accuracy defined, at what threshold, and under what production variation?
Why it matters: "99% detection rate" is meaningful only with context. Detection at what recall setting? Against what defect size? Under which lighting conditions?
- Weak answer: A single accuracy number from a controlled benchmark.
- Strong answer: Precision-recall curves across operating thresholds, tested against production-representative data including lighting variation, surface texture changes, and part orientation shifts. Also ask whether the system qualifies defects--distinguishing acceptable surface marks from ones that violate your quality specification--not just flags that a defect exists.
Ask for confusion matrices from a customer deployment in a comparable production environment--not from a lab dataset.
7. Can the model show which features triggered a defect flag?
Why it matters: For process engineers, a reject signal is only the beginning. Understanding why a defect was flagged--which region, which feature--is what enables root cause analysis and upstream process correction.
- Weak answer: "The model outputs a pass/fail label."
- Strong answer: Visual explainability (e.g., Grad-CAM heatmaps or bounding box overlays) that marks the defect region and, ideally, maps it to a defect category at micrometer-level precision.
HyperQ AI Vision surfaces visual explainability as a standard output, which manufacturing teams use to connect inspection results back to process adjustments on the line.
8. What is the retraining workflow when new defect types emerge post-deployment?
Why it matters: Production processes change. New materials, suppliers, or process shifts introduce defect types that didn't exist at deployment. Your vendor needs a defined, fast path for model updates.
- Weak answer: "Send us the new images and we'll retrain on our end"--opaque, slow, and creates vendor dependency.
- Strong answer: A customer-accessible labeling and retraining interface, with documented steps, expected timelines, and version control for deployed models.
Ask: Can your in-house team trigger and validate a retraining cycle without vendor involvement?
9. Is there a local support team in APAC, and what does the SLA look like?
Why it matters: For manufacturers operating in Southeast Asia, Japan, South Korea, or China, time-zone-aligned support is not optional. A production stoppage at 2 AM local time cannot wait for a US or European business day.
- Weak answer: "We have global support"--which typically means a shared ticket queue with 24--48 hour response windows.
- Strong answer: Named local engineers, defined response SLAs by severity tier (e.g., P1 line-down response within 2 hours), and evidence of deployed customers in your region.
Hypernology has APAC-based engineering support, which comes up consistently as a differentiator in competitive evaluations across Southeast Asian manufacturers.
10. Who are your reference customers, and in which industries and geographies?
Why it matters: A vendor's existing customer base tells you more than any demo. Industry-specific deployments signal genuine domain knowledge--not just computer vision capability applied generically.
- Weak answer: "We have customers in manufacturing" with no specifics.
- Strong answer: Named or describable reference customers in industries matching yours (electronics, automotive, food and beverage, medical devices), with deployed sites in your region, willing to take a reference call.
Ask for two references: one that is two-plus years into deployment, and one that went live in the last six months. The long-tenured customer tells you about stability and support; the recent one tells you about the current onboarding experience.
Using this framework in vendor comparisons
The questions above work as a scoring rubric. Assign each vendor a response quality rating--strong, partial, or weak--across all ten dimensions. Weight the categories by your operational priorities: if you're running high-mix lines, multi-SKU handling and retraining workflow deserve more weight. If you're in APAC with limited internal integration resources, deployment support and local SLAs move to the top.
The goal is not to find a vendor with perfect answers, but to surface honest answers--and to cut vendors who rely on vague claims where specificity is both possible and expected.
A structured evaluation using these ten questions typically separates commodity vision system vendors from those with genuine production-grade AI capability, regardless of brand recognition. Vendors who can answer in detail--with reference data, real customer names, and documented timelines--are the ones worth shortlisting.