# Engineering Safety in Machine Learning

arXiv: https://arxiv.org/abs/1601.04126 · HTML: https://mlsanity.com/paper/1601.04126/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jan 2016.

## Built on by (most cited first, 24 of 24)

- [Software Engineering for AI-Based Systems: A Survey](https://mlsanity.com/paper/2105.01984/) (2021)
- [Pitfalls of Epistemic Uncertainty Quantification through Loss Minimisation](https://mlsanity.com/paper/2203.06102/) (2022)
- [Is the Volume of a Credal Set a Good Measure for Epistemic Uncertainty?](https://mlsanity.com/paper/2306.09586/) (2023)
- [Second-Order Uncertainty Quantification: A Distance-Based Approach](https://mlsanity.com/paper/2312.00995/) (2023)
- [Taxonomy of Machine Learning Safety: A Survey and Primer](https://mlsanity.com/paper/2106.04823/) (2021)
- [Concrete Problems in AI Safety, Revisited](https://mlsanity.com/paper/2401.10899/) (2023)
- [Label-wise Aleatoric and Epistemic Uncertainty Quantification](https://mlsanity.com/paper/2406.02354/) (2024)
- [Second-Order Uncertainty Quantification: Variance-Based Measures](https://mlsanity.com/paper/2401.00276/) (2023)
- [On Out-of-distribution Detection with Energy-based Models](https://mlsanity.com/paper/2107.08785/) (2021)
- [Designing Fiduciary Artificial Intelligence](https://mlsanity.com/paper/2308.02435/) (2023)
- [Credal and Interval Deep Evidential Classifications](https://mlsanity.com/paper/2512.05526/) (2025)
- [Responsible and Regulatory Conform Machine Learning for Medicine: A Survey of Challenges and Solutions](https://mlsanity.com/paper/2107.09546/) (2021)
- [Verifiable Obstacle Detection](https://mlsanity.com/paper/2208.14403/) (2022)
- [RISE: Robust Individualized Decision Learning with Sensitive Variables](https://mlsanity.com/paper/2211.06569/) (2022)
- [Bringing a Ruler Into the Black Box: Uncovering Feature Impact from Individual Conditional Expectation Plots](https://mlsanity.com/paper/2109.02724/) (2021)
- [Delay-SDE-net: A deep learning approach for time series modelling with memory and uncertainty estimates](https://mlsanity.com/paper/2303.08587/) (2023)
- [Probabilistic Safety Regions Via Finite Families of Scalable Classifiers](https://mlsanity.com/paper/2309.04627/) (2023)
- [Understanding the Impact of Edge Cases from Occluded Pedestrians for ML Systems](https://mlsanity.com/paper/2204.12402/) (2022)
- [An LLM-Empowered Adaptive Evolutionary Algorithm For Multi-Component Deep Learning Systems](https://mlsanity.com/paper/2501.00829/) (2025)
- [Providing Error Detection for Deep Learning Image Classifiers Using Self-Explainability](https://mlsanity.com/paper/2210.08210/) (2022)
- [A Case Study on AI Engineering Practices: Developing an Autonomous Stock Trading System](https://mlsanity.com/paper/2303.13216/) (2023)
- [Enhancing Visual Perception in Novel Environments via Incremental Data Augmentation Based on Style Transfer](https://mlsanity.com/paper/2309.08851/) (2023)
- [Towards provable probabilistic safety for scalable embodied AI systems](https://mlsanity.com/paper/2506.05171/) (2025)
- [Adaptive Conformal Prediction via Bayesian Uncertainty Weighting for Hierarchical Healthcare Data](https://mlsanity.com/paper/2601.01223/) (2026)

## Abstract (arXiv)

Machine learning algorithms are increasingly influencing our decisions and interacting with us in all parts of our daily lives. Therefore, just like for power plants, highways, and myriad other engineered sociotechnical systems, we must consider the safety of systems involving machine learning. In this paper, we first discuss the definition of safety in terms of risk, epistemic uncertainty, and the harm incurred by unwanted outcomes. Then we examine dimensions, such as the choice of cost function and the appropriateness of minimizing the empirical average training cost, along which certain real-world applications may not be completely amenable to the foundational principle of modern statistical machine learning: empirical risk minimization. In particular, we note an emerging dichotomy of applications: ones in which safety is important and risk minimization is not the complete story (we name these Type A applications), and ones in which safety is not so critical and risk minimization is sufficient (we name these Type B applications). Finally, we discuss how four different strategies for achieving safety in engineering (inherently safe design, safety reserves, safe fail, and procedural safeguards) can be mapped to the machine learning context through interpretability and causality of predictive models, objectives beyond expected prediction accuracy, human involvement for labeling difficult or rare examples, and user experience design of software.