# Trimming the Risk: Towards Reliable Continuous Training for Deep Learning Inspection Systems

arXiv: https://arxiv.org/abs/2409.09108 · HTML: https://mlsanity.com/paper/2409.09108/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 36 of its 78 references that are in the map.
- First posted on arXiv 13 Sep 2024.

## Builds on (most cited first, 30 of 36)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [A Survey of Deep Active Learning](https://mlsanity.com/paper/2009.00236/) (2020)
- [Threat of Adversarial Attacks on Deep Learning in Computer Vision: A Survey](https://mlsanity.com/paper/1801.00553/) (2018)
- [Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples](https://mlsanity.com/paper/1711.09325/) (2017)
- [Snorkel: Rapid Training Data Creation with Weak Supervision](https://mlsanity.com/paper/1711.10160/) (2017)
- [Calibration of Pre-trained Transformers](https://mlsanity.com/paper/2003.07892/) (2020)
- [Mitigating Neural Network Overconfidence with Logit Normalization](https://mlsanity.com/paper/2205.09310/) (2022)
- [Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift](https://mlsanity.com/paper/1810.11953/) (2018)
- [Don't Blame the ELBO! A Linear VAE Perspective on Posterior Collapse](https://mlsanity.com/paper/1911.02469/) (2019)
- [Predicting with Confidence on Unseen Distributions](https://mlsanity.com/paper/2107.03315/) (2021)
- [Tilted Empirical Risk Minimization](https://mlsanity.com/paper/2007.01162/) (2020)
- [From Concept Drift to Model Degradation: An Overview on Performance-Aware Drift Detectors](https://mlsanity.com/paper/2203.11070/) (2022)
- [Classifier Calibration: A survey on how to assess and improve predicted class probabilities](https://mlsanity.com/paper/2112.10327/) (2021)
- [One-Class Classification: Taxonomy of Study and Review of Techniques](https://mlsanity.com/paper/1312.0049/) (2013)
- [A Survey on Programmatic Weak Supervision](https://mlsanity.com/paper/2202.05433/) (2022)
- [Understanding Softmax Confidence and Uncertainty](https://mlsanity.com/paper/2106.04972/) (2021)
- [Debiased Self-Training for Semi-Supervised Learning](https://mlsanity.com/paper/2202.07136/) (2022)
- [Post-hoc Uncertainty Calibration for Domain Drift Scenarios](https://mlsanity.com/paper/2012.10988/) (2020)
- [Snorkel DryBell: A Case Study in Deploying Weak Supervision at Industrial Scale](https://mlsanity.com/paper/1812.00417/) (2018)
- [Out-of-distribution Detection in Classifiers via Generation](https://mlsanity.com/paper/1910.04241/) (2019)
- [Affinity and Diversity: Quantifying Mechanisms of Data Augmentation](https://mlsanity.com/paper/2002.08973/) (2020)
- [A Learning Based Hypothesis Test for Harmful Covariate Shift](https://mlsanity.com/paper/2212.02742/) (2022)
- [DeepEvolution: A Search-Based Testing Approach for Deep Neural Networks](https://mlsanity.com/paper/1909.02563/) (2019)
- [Automatically detecting data drift in machine learning classifiers](https://mlsanity.com/paper/2111.05672/) (2021)
- [Towards out of distribution generalization for problems in mechanics](https://mlsanity.com/paper/2206.14917/) (2022)
- [Diffuse to Choose: Enriching Image Conditioned Inpainting in Latent Diffusion Models for Virtual Try-All](https://mlsanity.com/paper/2401.13795/) (2024)
- [CheXstray: Real-time Multi-Modal Data Concordance for Drift Detection in Medical Imaging AI](https://mlsanity.com/paper/2202.02833/) (2022)

## Abstract (arXiv)

The industry increasingly relies on deep learning (DL) technology for manufacturing inspections, which are challenging to automate with rule-based machine vision algorithms. DL-powered inspection systems derive defect patterns from labeled images, combining human-like agility with the consistency of a computerized system. However, finite labeled datasets often fail to encompass all natural variations necessitating Continuous Training (CT) to regularly adjust their models with recent data. Effective CT requires fresh labeled samples from the original distribution; otherwise, selfgenerated labels can lead to silent performance degradation. To mitigate this risk, we develop a robust CT-based maintenance approach that updates DL models using reliable data selections through a two-stage filtering process. The initial stage filters out low-confidence predictions, as the model inherently discredits them. The second stage uses variational auto-encoders and histograms to generate image embeddings that capture latent and pixel characteristics, then rejects the inputs of substantially shifted embeddings as drifted data with erroneous overconfidence. Then, a fine-tuning of the original DL model is executed on the filtered inputs while validating on a mixture of recent production and original datasets. This strategy mitigates catastrophic forgetting and ensures the model adapts effectively to new operational conditions. Evaluations on industrial inspection systems for popsicle stick prints and glass bottles using critical real-world datasets showed less than 9% of erroneous self-labeled data are retained after filtering and used for fine-tuning, improving model performance on production data by up to 14% without compromising its results on original validation data.