mlsanityData to 7 Oct 2026

How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Review

arXiv 2107.12045 ↗First posted 26 Jul 2021cs.LG cs.SEcited by 36

Not on the map. Why?

Abstract (arXiv)

Context: Machine Learning (ML) has been at the heart of many innovations over the past years. However, including it in so-called 'safety-critical' systems such as automotive or aeronautic has proven to be very challenging, since the shift in paradigm that ML brings completely changes traditional certification approaches. Objective: This paper aims to elucidate challenges related to the certification of ML-based safety-critical systems, as well as the solutions that are proposed in the literature to tackle them, answering the question 'How to Certify Machine Learning Based Safety-critical Systems?'. Method: We conduct a Systematic Literature Review (SLR) of research papers published between 2015 to 2020, covering topics related to the certification of ML systems. In total, we identified 217 papers covering topics considered to be the main pillars of ML certification: Robustness, Uncertainty, Explainability, Verification, Safe Reinforcement Learning, and Direct Certification. We analyzed the main trends and problems of each sub-field and provided summaries of the papers extracted. Results: The SLR results highlighted the enthusiasm of the community for this subject, as well as the lack of diversity in terms of datasets and type of models. It also emphasized the need to further develop connections between academia and industries to deepen the domain study. Finally, it also illustrated the necessity to build connections between the above mention main pillars that are for now mainly studied separately. Conclusion: We highlighted current efforts deployed to enable the certification of ML based software systems, and discuss some future research directions.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 160 of its 288 references that are in the map. First posted on arXiv 26 Jul 2021.

Builds on

160 of 288 references.

References · 160

  1. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  2. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  3. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  4. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  5. 2015Continuous control with deep reinforcement learningcited by 3,279
  6. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  7. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  8. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  9. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  10. 2019Explainable Artificial Intelligence (XAI): Concepts, Taxonomies, Opportunities and Challenges toward Responsible AIcited by 1,543
  11. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,472
  12. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,325
38 more
  1. 2017Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networkscited by 739
  2. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
  3. 2019Scaling Out-of-Distribution Detection for Real-World Settingscited by 465
  4. 2019Likelihood Ratios for Out-of-Distribution Detectioncited by 451
  5. 2020Graph Structure Learning for Robust Graph Neural Networkscited by 368
  6. 2017Safe Model-based Reinforcement Learning with Stability Guaranteescited by 347
  7. 2018Quantifying Generalization in Reinforcement Learningcited by 345
  8. 2016European Union regulations on algorithmic decision-making and a "right to explanation"cited by 343
  9. 2017Certifying Some Distributional Robustness with Principled Adversarial Trainingcited by 333
  10. 2018Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problemcited by 323
  11. 2017DeepXplore: Automated Whitebox Testing of Deep Learning Systemscited by 307
  12. 2017Certified Defenses for Data Poisoning Attackscited by 288
  13. 2016Safety Verification of Deep Neural Networkscited by 262
  14. 2019End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Taskscited by 256
  15. 2017DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Carscited by 249
  16. 2019Batch Policy Learning under Constraintscited by 242
  17. 2017DeepStack: Expert-Level Artificial Intelligence in No-Limit Pokercited by 228
  18. 2019Machine Learning Testing: Survey, Landscapes and Horizonscited by 213
  19. 2018An Introduction to Deep Reinforcement Learningcited by 201
  20. 2018Formal Security Analysis of Neural Networks using Symbolic Intervalscited by 185
  21. 2017Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulationcited by 182
  22. 2018Efficient Formal Safety Analysis of Neural Networkscited by 165
  23. 2017A General Safety Framework for Learning-Based Control in Uncertain Robotic Systemscited by 150
  24. 2018DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systemscited by 144
  25. 2019Branch and Bound for Piecewise Linear Neural Network Verificationcited by 138
  26. 2020NNV: The Neural Network Verification Tool for Deep Neural Networks and Learning-Enabled Cyber-Physical Systemscited by 128
  27. 2017On the Robustness of Semantic Segmentation Models to Adversarial Attackscited by 126
  28. 2018Parametric Noise Injection: Trainable Randomness to Improve Deep Neural Network Robustness against Adversarial Attackcited by 123
  29. 2019A General Framework for Uncertainty Estimation in Deep Learningcited by 119
  30. 2018A Survey on Data Collection for Machine Learning: a Big Data -- AI Integration Perspectivecited by 111
  31. 2020Multivariate Confidence Calibration for Object Detectioncited by 98
  32. 2020Training robust neural networks using Lipschitz boundscited by 98
  33. 2017Maximum Resilience of Artificial Neural Networkscited by 97
  34. 2018The Lyapunov Neural Network: Adaptive Stability Certification for Safe Learning of Dynamical Systemscited by 87
  35. 2019Evaluating and Calibrating Uncertainty Prediction in Regression Taskscited by 85
  36. 2018Omega-Regular Objectives in Model-Free Reinforcement Learningcited by 74
  37. 2019Sampling-free Epistemic Uncertainty Estimation Using Approximated Variance Propagationcited by 74
  38. 2016Safe Exploration in Finite Markov Decision Processes with Gaussian Processescited by 72

50 most cited of 160 All 160 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 5; 2023: 11; 2024: 6; 2025: 5; 2026: 8.

36 papers.

Sort:

By area:

  1. 2021Safe Learning in Robotics: From Learning-Based Control to Safe Reinforcement Learningcited by 337
  2. 2022Hardware Approximate Techniques for Deep Neural Network Accelerators: A Surveycited by 39
  3. 2023METAL: Metamorphic Testing Framework for Analyzing Large-Language Model Qualitiescited by 20
  4. 2024Distributionally Robust Constrained Reinforcement Learning under Strong Dualitycited by 15
  5. 2022Ergo, SMIRK is Safe: A Safety Case for a Machine Learning Component in a Pedestrian Automatic Emergency Brake Systemcited by 14
  6. 2023DeepGD: A Multi-Objective Black-Box Test Selection Approach for Deep Neural Networkscited by 11
  7. 2023Memory Efficient Neural Processes via Constant Memory Attention Blockcited by 9
  8. 2023Data Cleaning and Machine Learning: A Systematic Literature Reviewcited by 9
  9. 2025Probabilistic Robustness in Deep Learning: A Concise yet Comprehensive Guidecited by 6
  10. 2023Optimality Principles in Spacecraft Neural Guidance and Controlcited by 4
  11. 2022A Probabilistic Framework for Mutation Testing in Deep Neural Networkscited by 3
  12. 2026A Reference Architecture of Reinforcement Learning Frameworkscited by 3
24 more
  1. 2023Delay-SDE-net: A deep learning approach for time series modelling with memory and uncertainty estimatescited by 2
  2. 2024Forward Learning for Gradient-based Black-box Saliency Map Generationcited by 2
  3. 2024Operational range bounding of spectroscopy models with anomaly detectioncited by 2
  4. 2025Out-of-Distribution Detection for Safety Assurance of AI and Autonomous Systemscited by 2
  5. 2022Perception Simplex: Verifiable Collision Avoidance in Autonomous Vehicles Amidst Obstacle Detection Faultscited by 1
  6. 2023Verifying Safety of Neural Networks from Topological Perspectivescited by 1
  7. 2023GIST: Generated Inputs Sets Transferability in Deep Learningcited by 1
  8. 2023Modeling Risk in Reinforcement Learning: A Literature Mappingcited by 1
  9. 2025CoCoAFusE: Beyond Mixtures of Experts via Model Fusioncited by 1
  10. 2026Defining Operational Conditions for Safety-Critical AI-Based Systems from Datacited by 1
  11. 2022Safe AI -- How is this Possible?cited by 0
  12. 2023SEE-OoD: Supervised Exploration For Enhanced Out-of-Distribution Detectioncited by 0
  13. 2023Learning Polynomial Representations of Physical Objects with Application to Certifying Correct Packing Configurationscited by 0
  14. 2024Testing Spintronics Implemented Monte Carlo Dropout-Based Bayesian Neural Networkscited by 0
  15. 2024Science based AI model certification for untrained operational environments with application in traffic state estimationcited by 0
  16. 2024Science based AI model certification for new operational environments with application in traffic state estimationcited by 0
  17. 2025Fixed Point Explainabilitycited by 0
  18. 2025The Engineer's Dilemma: A Review of Establishing a Legal Framework for Integrating Machine Learning in Construction by Navigating Precedents and Industry Expectationscited by 0
  19. 2026VOLTA: The Surprising Ineffectiveness of Auxiliary Losses for Calibrated Deep Learningcited by 0
  20. 2026Probabilistic Verification of Neural Networks via Efficient Probabilistic Hull Generationcited by 0
  21. 2026Laplace-Bridged Randomized Smoothing for Fast Certified Robustnesscited by 0
  22. 2026Class Imbalance and Batch Effects in LLM-Based Screening for Systematic Reviewscited by 0
  23. 2026On the Applicability of Safety Nets: A Safety-By-Design Solution for Certifying Neural Networkscited by 0
  24. 2026Operational Range Bounding in Spectroscopy: A Safety Cage Framework for Machine Learning Modelscited by 0