R&D
CENTER
KOENJA

Model Evaluation and Field Operations: False Rejects, Escapes, Thresholds, Retraining

A 99% accuracy number alone is not enough to put a model on the line. This lesson explains how to count false rejects and escapes separately, set the threshold to match cost, and monitor changes in operation so you can retrain.

Core4 min0 views
Key points
  1. A false reject (overkill: a good part flagged as defective) and an escape (a missed defect) have different costs. Count them separately.
  2. Set the threshold not to maximize accuracy, but to keep escapes within the allowed limit while reducing false rejects.
  3. Use a separate evaluation set that was not used for training or threshold tuning. Only then can you trust the numbers.
  4. In operation, monitor changes in the score distribution (drift) and build a loop that feeds field decisions back into training.

Confusion Matrix: Count in Four Cells

Actually good Actually defective
Model: OK Correct Escape (missed defect; a defect gets out)
Model: NG False reject (overkill; a good part is scrapped or re-inspected) Correct

On a line with a 1% defect rate, a "model" that judges every part OK has 99% accuracy. Yet it catches no defects at all. So look at the metrics below together, not at accuracy alone.

  • Recall — The share of actual defects the model catches. The fewer the escapes, the higher it is.
  • Precision — The share of the model's NG calls that are real defects. The fewer the false rejects, the higher it is.
  • F1 — The harmonic mean of recall and precision. Use it to see the balance of the two as one number.

Set the Threshold by Cost

Good part scores Defective part scores threshold Escape False reject 01
In the zone where the two distributions overlap, moving the threshold left reduces escapes and increases false rejects. Removing the overlap is the job of the image and the model, not the threshold.

On most lines, one escape costs far more than one false reject. A defect that reaches the customer leads to claims and recalls, while a false reject costs a re-inspection. So the order is:

  1. Set the escape rate you can allow first. Example: 0 cases.
  2. Among the thresholds that meet that condition, pick the one with the fewest false rejects.
  3. If false rejects are still too many, improve the image (lighting, alignment) and the model, not the threshold.

RoboVision has a feature that automatically finds the point with the highest F1. It is a good starting point, but adjust the final value against this cost criterion. Hand zones where the model is unsure to a person through the Gray zone of the dual threshold.

Split the Data into Three Sets

  • Training set — The data the model learns from.
  • Validation set — The data you look at when choosing the algorithm and threshold.
  • Evaluation set — The data you use only once, at the end, to check performance.

If you tune the threshold while looking at the evaluation set, that number comes out better than reality. Also, do not let images of the same part taken from several angles end up split between training and evaluation. That would mean the model re-identifies a part it has already seen.

Putting It on the Line

  • Speed — The decision must finish within the line takt time. Measure not only the inference time for one image but the total including capture, preprocessing and result transfer.
  • Optimized format — Converting a trained model to ONNX or TensorRT speeds up inference on the same GPU. The Model Hub of the Physical AI Platform (PaaS) manages conversion and deployment.
  • Result logging — Save the decision, score and image path as CSV and collect them in fineAi Dash. You can then see trends by time slot, line and defect type, plus SPC and Pareto charts.

Change in Operation: Drift

A line keeps changing. Lights age, raw material lots change, and cameras shift slightly. Then the score distribution of good parts moves little by little to the right, and one day false rejects suddenly rise. PaaS computes how far the score distribution has moved from training time as the PSI (Population Stability Index) and raises a warning. You can find the cause before false rejects spike.

The Retraining Loop

  1. Field operators mark the real decision on NG and Gray images.
  2. The marked images accumulate in the dataset.
  3. Retrain periodically, and check with the evaluation set that the new model beats the previous one.
  4. If it does, deploy it to field equipment by OTA.

When this loop runs, the model improves along with the line. Without the loop, performance on release day is the peak.