# On The Reliability Of Machine Learning Applications In Manufacturing Environments

arXiv: https://arxiv.org/abs/2112.06986 · HTML: https://mlsanity.com/paper/2112.06986/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 29 references that are in the map.
- First posted on arXiv 13 Dec 2021.

## Builds on (most cited first, 9 of 9)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [Continual Learning in Practice](https://mlsanity.com/paper/1903.05202/) (2019)
- [Detecting Concept Drift With Neural Network Model Uncertainty](https://mlsanity.com/paper/2107.01873/) (2021)
- [Unknown Examples & Machine Learning Model Generalization](https://mlsanity.com/paper/1808.08294/) (2018)
- [Ensuring the Robustness and Reliability of Data-Driven Knowledge Discovery Models in Production and Manufacturing](https://mlsanity.com/paper/2007.14791/) (2020)

## Built on by (most cited first, 3 of 3)

- [Generative Machine Learning in Adaptive Control of Dynamic Manufacturing Processes: A Review](https://mlsanity.com/paper/2505.00210/) (2025)
- [Conformal Prediction in Multi-User Settings: An Evaluation](https://mlsanity.com/paper/2312.05195/) (2023)
- [A temporal deep learning framework for calibration of low-cost air quality sensors](https://mlsanity.com/paper/2604.21527/) (2026)

## Abstract (arXiv)

The increasing deployment of advanced digital technologies such as Internet of Things (IoT) devices and Cyber-Physical Systems (CPS) in industrial environments is enabling the productive use of machine learning (ML) algorithms in the manufacturing domain. As ML applications transcend from research to productive use in real-world industrial environments, the question of reliability arises. Since the majority of ML models are trained and evaluated on static datasets, continuous online monitoring of their performance is required to build reliable systems. Furthermore, concept and sensor drift can lead to degrading accuracy of the algorithm over time, thus compromising safety, acceptance and economics if undetected and not properly addressed. In this work, we exemplarily highlight the severity of the issue on a publicly available industrial dataset which was recorded over the course of 36 months and explain possible sources of drift. We assess the robustness of ML algorithms commonly used in manufacturing and show, that the accuracy strongly declines with increasing drift for all tested algorithms. We further investigate how uncertainty estimation may be leveraged for online performance estimation as well as drift detection as a first step towards continually learning applications. The results indicate, that ensemble algorithms like random forests show the least decay of confidence calibration under drift.