# How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Review

arXiv: https://arxiv.org/abs/2107.12045 · HTML: https://mlsanity.com/paper/2107.12045/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 160 of its 288 references that are in the map.
- First posted on arXiv 26 Jul 2021.

## Builds on (most cited first, 30 of 160)

- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AI](https://mlsanity.com/paper/1910.10045/) (2019)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks](https://mlsanity.com/paper/1702.01135/) (2017)
- [Opening the Black Box of Deep Neural Networks via Information](https://mlsanity.com/paper/1703.00810/) (2017)
- [Scaling Out-of-Distribution Detection for Real-World Settings](https://mlsanity.com/paper/1911.11132/) (2019)
- [Likelihood Ratios for Out-of-Distribution Detection](https://mlsanity.com/paper/1906.02845/) (2019)
- [Graph Structure Learning for Robust Graph Neural Networks](https://mlsanity.com/paper/2005.10203/) (2020)
- [Safe Model-based Reinforcement Learning with Stability Guarantees](https://mlsanity.com/paper/1705.08551/) (2017)
- [Quantifying Generalization in Reinforcement Learning](https://mlsanity.com/paper/1812.02341/) (2018)
- [European Union regulations on algorithmic decision-making and a "right to explanation"](https://mlsanity.com/paper/1606.08813/) (2016)
- [Certifying Some Distributional Robustness with Principled Adversarial Training](https://mlsanity.com/paper/1710.10571/) (2017)
- [Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problem](https://mlsanity.com/paper/1812.05720/) (2018)
- [DeepXplore: Automated Whitebox Testing of Deep Learning Systems](https://mlsanity.com/paper/1705.06640/) (2017)
- [Certified Defenses for Data Poisoning Attacks](https://mlsanity.com/paper/1706.03691/) (2017)
- [End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Tasks](https://mlsanity.com/paper/1903.08792/) (2019)
- [Safety Verification of Deep Neural Networks](https://mlsanity.com/paper/1610.06940/) (2016)
- [DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars](https://mlsanity.com/paper/1708.08559/) (2017)
- [Batch Policy Learning under Constraints](https://mlsanity.com/paper/1903.08738/) (2019)
- [DeepStack: Expert-Level Artificial Intelligence in No-Limit Poker](https://mlsanity.com/paper/1701.01724/) (2017)
- [Machine Learning Testing: Survey, Landscapes and Horizons](https://mlsanity.com/paper/1906.10742/) (2019)

## Built on by (most cited first, 30 of 39)

- [Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learning](https://mlsanity.com/paper/2108.06266/) (2021)
- [Hardware Approximate Techniques for Deep Neural Network Accelerators: A Survey](https://mlsanity.com/paper/2203.08737/) (2022)
- [METAL: Metamorphic Testing Framework for Analyzing Large-Language Model Qualities](https://mlsanity.com/paper/2312.06056/) (2023)
- [Distributionally Robust Constrained Reinforcement Learning under Strong Duality](https://mlsanity.com/paper/2406.15788/) (2024)
- [Ergo, SMIRK is Safe: A Safety Case for a Machine Learning Component in a Pedestrian Automatic Emergency Brake System](https://mlsanity.com/paper/2204.07874/) (2022)
- [DeepGD: A Multi-Objective Black-Box Test Selection Approach for Deep Neural Networks](https://mlsanity.com/paper/2303.04878/) (2023)
- [Memory Efficient Neural Processes via Constant Memory Attention Block](https://mlsanity.com/paper/2305.14567/) (2023)
- [Data Cleaning and Machine Learning: A Systematic Literature Review](https://mlsanity.com/paper/2310.01765/) (2023)
- [Probabilistic Robustness in Deep Learning: A Concise yet Comprehensive Guide](https://mlsanity.com/paper/2502.14833/) (2025)
- [Optimality Principles in Spacecraft Neural Guidance and Control](https://mlsanity.com/paper/2305.13078/) (2023)
- [A Probabilistic Framework for Mutation Testing in Deep Neural Networks](https://mlsanity.com/paper/2208.06018/) (2022)
- [A Reference Architecture of Reinforcement Learning Frameworks](https://mlsanity.com/paper/2603.06413/) (2026)
- [Delay-SDE-net: A deep learning approach for time series modelling with memory and uncertainty estimates](https://mlsanity.com/paper/2303.08587/) (2023)
- [Forward Learning for Gradient-based Black-box Saliency Map Generation](https://mlsanity.com/paper/2403.15603/) (2024)
- [Operational range bounding of spectroscopy models with anomaly detection](https://mlsanity.com/paper/2408.02581/) (2024)
- [Out-of-Distribution Detection for Safety Assurance of AI and Autonomous Systems](https://mlsanity.com/paper/2510.21254/) (2025)
- [Perception Simplex: Verifiable Collision Avoidance in Autonomous Vehicles Amidst Obstacle Detection Faults](https://mlsanity.com/paper/2209.01710/) (2022)
- [Verifying Safety of Neural Networks from Topological Perspectives](https://mlsanity.com/paper/2306.15403/) (2023)
- [GIST: Generated Inputs Sets Transferability in Deep Learning](https://mlsanity.com/paper/2311.00801/) (2023)
- [Modeling Risk in Reinforcement Learning: A Literature Mapping](https://mlsanity.com/paper/2312.05231/) (2023)
- [CoCoAFusE: Beyond Mixtures of Experts via Model Fusion](https://mlsanity.com/paper/2505.01105/) (2025)
- [Control Synthesis with Reinforcement Learning: A Modeling Perspective](https://mlsanity.com/paper/2510.25063/) (2025)
- [Defining Operational Conditions for Safety-Critical AI-Based Systems from Data](https://mlsanity.com/paper/2601.22118/) (2026)
- [Safe AI -- How is this Possible?](https://mlsanity.com/paper/2201.10436/) (2022)
- [SEE-OoD: Supervised Exploration For Enhanced Out-of-Distribution Detection](https://mlsanity.com/paper/2310.08040/) (2023)
- [Learning Polynomial Representations of Physical Objects with Application to Certifying Correct Packing Configurations](https://mlsanity.com/paper/2312.06791/) (2023)
- [Testing Spintronics Implemented Monte Carlo Dropout-Based Bayesian Neural Networks](https://mlsanity.com/paper/2401.04744/) (2024)
- [Science based AI model certification for untrained operational environments with application in traffic state estimation](https://mlsanity.com/paper/2403.14093/) (2024)
- [Science based AI model certification for new operational environments with application in traffic state estimation](https://mlsanity.com/paper/2405.07893/) (2024)
- [Bayes2IMC: In-Memory Computing for Bayesian Binary Neural Networks](https://mlsanity.com/paper/2411.07902/) (2024)

## Abstract (arXiv)

Context: Machine Learning (ML) has been at the heart of many innovations over the past years. However, including it in so-called 'safety-critical' systems such as automotive or aeronautic has proven to be very challenging, since the shift in paradigm that ML brings completely changes traditional certification approaches. Objective: This paper aims to elucidate challenges related to the certification of ML-based safety-critical systems, as well as the solutions that are proposed in the literature to tackle them, answering the question 'How to Certify Machine Learning Based Safety-critical Systems?'. Method: We conduct a Systematic Literature Review (SLR) of research papers published between 2015 to 2020, covering topics related to the certification of ML systems. In total, we identified 217 papers covering topics considered to be the main pillars of ML certification: Robustness, Uncertainty, Explainability, Verification, Safe Reinforcement Learning, and Direct Certification. We analyzed the main trends and problems of each sub-field and provided summaries of the papers extracted. Results: The SLR results highlighted the enthusiasm of the community for this subject, as well as the lack of diversity in terms of datasets and type of models. It also emphasized the need to further develop connections between academia and industries to deepen the domain study. Finally, it also illustrated the necessity to build connections between the above mention main pillars that are for now mainly studied separately. Conclusion: We highlighted current efforts deployed to enable the certification of ML based software systems, and discuss some future research directions.