# Can Search-Based Testing with Pareto Optimization Effectively Cover Failure-Revealing Test Inputs?

arXiv: https://arxiv.org/abs/2410.11769 · HTML: https://mlsanity.com/paper/2410.11769/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 66 references that are in the map.
- First posted on arXiv 15 Oct 2024.

## Builds on (most cited first, 22 of 22)

- [Illuminating search spaces by mapping elites](https://mlsanity.com/paper/1504.04909/) (2015)
- [Model-based Exploration of the Frontier of Behaviours for Deep Learning System Testing](https://mlsanity.com/paper/2007.02787/) (2020)
- [DeepHyperion: Exploring the Feature Space of Deep Learning-Based Systems through Illumination Search](https://mlsanity.com/paper/2107.06997/) (2021)
- [Black-Box Testing of Deep Neural Networks Through Test Case Diversity](https://mlsanity.com/paper/2112.12591/) (2021)
- [When and Why Test Generators for Deep Learning Produce Invalid Inputs: an Empirical Study](https://mlsanity.com/paper/2212.11368/) (2022)
- [How to Evaluate Solutions in Pareto-based Search-Based Software Engineering? A Critical Review and Methodological Guidance](https://mlsanity.com/paper/2002.09040/) (2020)
- [Digital Twins Are Not Monozygotic -- Cross-Replicating ADAS Testing in Two Industry-Grade Automotive Simulators](https://mlsanity.com/paper/2012.06822/) (2020)
- [Test Set Diameter: Quantifying the Diversity of Sets of Test Cases](https://mlsanity.com/paper/1506.03482/) (2015)
- [Identifying and Explaining Safety-critical Scenarios for Autonomous Vehicles via Key Features](https://mlsanity.com/paper/2212.07566/) (2022)
- [AmbieGen: A Search-based Framework for Autonomous Systems Testing](https://mlsanity.com/paper/2301.01234/) (2023)
- [Efficient and Effective Generation of Test Cases for Pedestrian Detection -- Search-based Software Testing of Baidu Apollo in SVL](https://mlsanity.com/paper/2109.07960/) (2021)
- [Single and Multi-objective Test Cases Prioritization for Self-driving Cars in Virtual Environments](https://mlsanity.com/paper/2107.09614/) (2021)
- [A Search-Based Framework for Automatic Generation of Testing Environments for Cyber-Physical Systems](https://mlsanity.com/paper/2203.12138/) (2022)
- [Towards Reliable AI: Adequacy Metrics for Ensuring the Quality of System-level Testing of Autonomous Vehicles](https://mlsanity.com/paper/2311.08049/) (2023)
- [Risk-Aware Scene Sampling for Dynamic Assurance of Autonomous Systems](https://mlsanity.com/paper/2202.13510/) (2022)
- [Machine Learning Testing in an ADAS Case Study Using Simulation-Integrated Bio-Inspired Search-Based Testing](https://mlsanity.com/paper/2203.12026/) (2022)
- [OpenSBT: A Modular Framework for Search-based Testing of Automated Driving Systems](https://mlsanity.com/paper/2306.10296/) (2023)
- [Reinforcement Learning Informed Evolutionary Search for Autonomous Systems Testing](https://mlsanity.com/paper/2308.12762/) (2023)
- [Test Generation Strategies for Building Failure Models and Explaining Spurious Failures](https://mlsanity.com/paper/2312.05631/) (2023)
- [Reflections on Surrogate-Assisted Search-Based Testing: A Taxonomy and Two Replication Studies based on Industrial ADAS and Simulink Models](https://mlsanity.com/paper/2305.00083/) (2023)
- [Guiding the Search Towards Failure-Inducing Test Inputs Using Support Vector Machines](https://mlsanity.com/paper/2401.12364/) (2024)
- [Searching for test data with feature diversity](https://mlsanity.com/paper/1709.06017/) (2017)

## Built on by (most cited first, 1 of 1)

- [Simulator Ensembles for Trustworthy Autonomous Driving Systems Testing](https://mlsanity.com/paper/2503.08936/) (2025)

## Abstract (arXiv)

Search-based software testing (SBST) is a widely adopted technique for testing complex systems with large input spaces, such as Deep Learning-enabled (DL-enabled) systems. Many SBST techniques focus on Pareto-based optimization, where multiple objectives are optimized in parallel to reveal failures. However, it is important to ensure that identified failures are spread throughout the entire failure-inducing area of a search domain and not clustered in a sub-region. This ensures that identified failures are semantically diverse and reveal a wide range of underlying causes. In this paper, we present a theoretical argument explaining why testing based on Pareto optimization is inadequate for covering failure-inducing areas within a search domain. We support our argument with empirical results obtained by applying two widely used types of Pareto-based optimization techniques, namely NSGA-II (an evolutionary algorithm) and OMOPSO (a swarm-based Pareto-optimization algorithm), to two DL-enabled systems: an industrial Automated Valet Parking (AVP) system and a system for classifying handwritten digits. We measure the coverage of failure-revealing test inputs in the input space using a metric that we refer to as the Coverage Inverted Distance quality indicator. Our results show that NSGA-II-based search and OMOPSO are not more effective than a na"ive random search baseline in covering test inputs that reveal failures. The replication package for this study is available in a GitHub repository.