# Machine Learning Testing in an ADAS Case Study Using Simulation-Integrated Bio-Inspired Search-Based Testing

arXiv: https://arxiv.org/abs/2203.12026 · HTML: https://mlsanity.com/paper/2203.12026/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 62 references that are in the map.
- First posted on arXiv 22 Mar 2022.

## Builds on (most cited first, 15 of 15)

- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [End to End Learning for Self-Driving Cars](https://mlsanity.com/paper/1604.07316/) (2016)
- [DeepXplore: Automated Whitebox Testing of Deep Learning Systems](https://mlsanity.com/paper/1705.06640/) (2017)
- [DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars](https://mlsanity.com/paper/1708.08559/) (2017)
- [LGSVL Simulator: A High Fidelity Simulator for Autonomous Driving](https://mlsanity.com/paper/2005.03778/) (2020)
- [Guidance on the Assurance of Machine Learning in Autonomous Systems (AMLAS)](https://mlsanity.com/paper/2102.01564/) (2021)
- [Model-based Exploration of the Frontier of Behaviours for Deep Learning System Testing](https://mlsanity.com/paper/2007.02787/) (2020)
- [Safely Entering the Deep: A Review of Verification and Validation for Machine Learning and a Challenge Elicitation in the Automotive Industry](https://mlsanity.com/paper/1812.05389/) (2018)
- [A Survey on Simulators for Testing Self-Driving Cars](https://mlsanity.com/paper/2101.05337/) (2021)
- [Can Offline Testing of Deep Neural Networks Replace Their Online Testing?](https://mlsanity.com/paper/2101.11118/) (2021)
- [Digital Twins Are Not Monozygotic -- Cross-Replicating ADAS Testing in Two Industry-Grade Automotive Simulators](https://mlsanity.com/paper/2012.06822/) (2020)
- [Automatic Test Suite Generation for Key-Points Detection DNNs using Many-Objective Search (Experience Paper)](https://mlsanity.com/paper/2012.06511/) (2020)
- [Efficient and Effective Generation of Test Cases for Pedestrian Detection -- Search-based Software Testing of Baidu Apollo in SVL](https://mlsanity.com/paper/2109.07960/) (2021)
- [Exploring the Assessment List for Trustworthy AI in the Context of Advanced Driver-Assistance Systems](https://mlsanity.com/paper/2103.09051/) (2021)
- [Single and Multi-objective Test Cases Prioritization for Self-driving Cars in Virtual Environments](https://mlsanity.com/paper/2107.09614/) (2021)

## Built on by (most cited first, 4 of 4)

- [Two is Better Than One: Digital Siblings to Improve Autonomous Driving Testing](https://mlsanity.com/paper/2305.08060/) (2023)
- [Bridging Data-Driven and Knowledge-Driven Approaches for Safety-Critical Scenario Generation in Automated Vehicle Validation](https://mlsanity.com/paper/2311.10937/) (2023)
- [Can Search-Based Testing with Pareto Optimization Effectively Cover Failure-Revealing Test Inputs?](https://mlsanity.com/paper/2410.11769/) (2024)
- [Simulator Ensembles for Trustworthy Autonomous Driving Systems Testing](https://mlsanity.com/paper/2503.08936/) (2025)

## Abstract (arXiv)

This paper presents an extended version of Deeper, a search-based simulation-integrated test solution that generates failure-revealing test scenarios for testing a deep neural network-based lane-keeping system. In the newly proposed version, we utilize a new set of bio-inspired search algorithms, genetic algorithm (GA), (μ+λ) and (μ,λ) evolution strategies (ES), and particle swarm optimization (PSO), that leverage a quality population seed and domain-specific cross-over and mutation operations tailored for the presentation model used for modeling the test scenarios. In order to demonstrate the capabilities of the new test generators within Deeper, we carry out an empirical evaluation and comparison with regard to the results of five participating tools in the cyber-physical systems testing competition at SBST 2021. Our evaluation shows the newly proposed test generators in Deeper not only represent a considerable improvement on the previous version but also prove to be effective and efficient in provoking a considerable number of diverse failure-revealing test scenarios for testing an ML-driven lane-keeping system. They can trigger several failures while promoting test scenario diversity, under a limited test time budget, high target failure severity, and strict speed limit constraints.