# Safely Entering the Deep: A Review of Verification and Validation for Machine Learning and a Challenge Elicitation in the Automotive Industry

arXiv: https://arxiv.org/abs/1812.05389 · HTML: https://mlsanity.com/paper/1812.05389/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Dec 2018.

## Built on by (most cited first, 30 of 38)

- [Explainability of deep vision-based autonomous driving systems: Review and challenges](https://mlsanity.com/paper/2101.05307/) (2021)
- [Software Engineering for AI-Based Systems: A Survey](https://mlsanity.com/paper/2105.01984/) (2021)
- [Collaboration Challenges in Building ML-Enabled Systems: Communication, Documentation, Engineering, and Process](https://mlsanity.com/paper/2110.10234/) (2021)
- [A Meta-Summary of Challenges in Building Products with ML Components -- Collecting Experiences from 4758+ Practitioners](https://mlsanity.com/paper/2304.00078/) (2023)
- [Ergo, SMIRK is Safe: A Safety Case for a Machine Learning Component in a Pedestrian Automatic Emergency Brake System](https://mlsanity.com/paper/2204.07874/) (2022)
- [Increasing the Confidence of Deep Neural Networks by Coverage Analysis](https://mlsanity.com/paper/2101.12100/) (2021)
- [A Review of Testing Object-Based Environment Perception for Safe Automated Driving](https://mlsanity.com/paper/2102.08460/) (2021)
- [Performance Analysis of Out-of-Distribution Detection on Various Trained Neural Networks](https://mlsanity.com/paper/2103.15580/) (2021)
- [Efficient and Effective Generation of Test Cases for Pedestrian Detection -- Search-based Software Testing of Baidu Apollo in SVL](https://mlsanity.com/paper/2109.07960/) (2021)
- [Responsible and Regulatory Conform Machine Learning for Medicine: A Survey of Challenges and Solutions](https://mlsanity.com/paper/2107.09546/) (2021)
- [Performance Analysis of Out-of-Distribution Detection on Trained Neural Networks](https://mlsanity.com/paper/2204.12378/) (2022)
- [Requirements Engineering for Machine Learning: A Review and Reflection](https://mlsanity.com/paper/2210.00859/) (2022)
- [Exploring the Assessment List for Trustworthy AI in the Context of Advanced Driver-Assistance Systems](https://mlsanity.com/paper/2103.09051/) (2021)
- [Requirement Engineering Challenges for AI-intense Systems Development](https://mlsanity.com/paper/2103.10270/) (2021)
- [Maximum Likelihood Uncertainty Estimation: Robustness to Outliers](https://mlsanity.com/paper/2202.03870/) (2022)
- [Toward Certification of Machine-Learning Systems for Low Criticality Airborne Applications](https://mlsanity.com/paper/2209.13975/) (2022)
- [Evaluation of Out-of-Distribution Detection Performance on Autonomous Driving Datasets](https://mlsanity.com/paper/2401.17013/) (2024)
- [Artificial Intelligence in Industry 4.0: A Review of Integration Challenges for Industrial Systems](https://mlsanity.com/paper/2405.18580/) (2024)
- [Agility in Software 2.0 -- Notebook Interfaces and MLOps with Buttresses and Rebars](https://mlsanity.com/paper/2111.14142/) (2021)
- [Machine Learning Testing in an ADAS Case Study Using Simulation-Integrated Bio-Inspired Search-Based Testing](https://mlsanity.com/paper/2203.12026/) (2022)
- [An investigation of challenges encountered when specifying training data and runtime monitors for safety critical ML applications](https://mlsanity.com/paper/2301.13476/) (2023)
- [Automotive Perception Software Development: An Empirical Investigation into Data, Annotation, and Ecosystem Challenges](https://mlsanity.com/paper/2303.05947/) (2023)
- [Test Automation with Grad-CAM Heatmaps -- A Future Pipe Segment in MLOps for Vision AI?](https://mlsanity.com/paper/2103.01837/) (2021)
- [Exploring ML testing in practice -- Lessons learned from an interactive rapid review with Axis Communications](https://mlsanity.com/paper/2203.16225/) (2022)
- [Reward Bound for Behavioral Guarantee of Model-based Planning Agents](https://mlsanity.com/paper/2402.13419/) (2024)
- [Safety Monitoring of Machine Learning Perception Functions: a Survey](https://mlsanity.com/paper/2412.06869/) (2024)
- [MLTEing Models: Negotiating, Evaluating, and Documenting Model and System Qualities](https://mlsanity.com/paper/2303.01998/) (2023)
- [Forward Learning for Gradient-based Black-box Saliency Map Generation](https://mlsanity.com/paper/2403.15603/) (2024)
- [Operational range bounding of spectroscopy models with anomaly detection](https://mlsanity.com/paper/2408.02581/) (2024)
- [Weakly Supervised Reinforcement Learning for Autonomous Highway Driving via Virtual Safety Cages](https://mlsanity.com/paper/2103.09726/) (2021)

## Abstract (arXiv)

Deep Neural Networks (DNN) will emerge as a cornerstone in automotive software engineering. However, developing systems with DNNs introduces novel challenges for safety assessments. This paper reviews the state-of-the-art in verification and validation of safety-critical systems that rely on machine learning. Furthermore, we report from a workshop series on DNNs for perception with automotive experts in Sweden, confirming that ISO 26262 largely contravenes the nature of DNNs. We recommend aerospace-to-automotive knowledge transfer and systems-based safety approaches, e.g., safety cage architectures and simulated system test cases.