Can Search-Based Testing with Pareto Optimization Effectively Cover Failure-Revealing Test Inputs?
Not on the map. Why?
Abstract (arXiv)
Search-based software testing (SBST) is a widely adopted technique for testing complex systems with large input spaces, such as Deep Learning-enabled (DL-enabled) systems. Many SBST techniques focus on Pareto-based optimization, where multiple objectives are optimized in parallel to reveal failures. However, it is important to ensure that identified failures are spread throughout the entire failure-inducing area of a search domain and not clustered in a sub-region. This ensures that identified failures are semantically diverse and reveal a wide range of underlying causes. In this paper, we present a theoretical argument explaining why testing based on Pareto optimization is inadequate for covering failure-inducing areas within a search domain. We support our argument with empirical results obtained by applying two widely used types of Pareto-based optimization techniques, namely NSGA-II (an evolutionary algorithm) and OMOPSO (a swarm-based Pareto-optimization algorithm), to two DL-enabled systems: an industrial Automated Valet Parking (AVP) system and a system for classifying handwritten digits. We measure the coverage of failure-revealing test inputs in the input space using a metric that we refer to as the Coverage Inverted Distance quality indicator. Our results show that NSGA-II-based search and OMOPSO are not more effective than a na"ive random search baseline in covering test inputs that reveal failures. The replication package for this study is available in a GitHub repository.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 22 of its 66 references that are in the map. First posted on arXiv 15 Oct 2024.
Builds on
22 of 66 references.
References · 22
- 2015Illuminating search spaces by mapping elitescited by 406
- 2020Model-based Exploration of the Frontier of Behaviours for Deep Learning System Testingcited by 40
- 2021DeepHyperion: Exploring the Feature Space of Deep Learning-Based Systems through Illumination Searchcited by 29
- 2021Black-Box Testing of Deep Neural Networks Through Test Case Diversitycited by 18
- 2022When and Why Test Generators for Deep Learning Produce Invalid Inputs: an Empirical Studycited by 18
- 2020How to Evaluate Solutions in Pareto-based Search-Based Software Engineering? A Critical Review and Methodological Guidancecited by 17
- 2020Digital Twins Are Not Monozygotic -- Cross-Replicating ADAS Testing in Two Industry-Grade Automotive Simulatorscited by 12
- 2015Test Set Diameter: Quantifying the Diversity of Sets of Test Casescited by 10
- 2022Identifying and Explaining Safety-critical Scenarios for Autonomous Vehicles via Key Featurescited by 9
- 2021Efficient and Effective Generation of Test Cases for Pedestrian Detection -- Search-based Software Testing of Baidu Apollo in SVLcited by 8
- 2023AmbieGen: A Search-based Framework for Autonomous Systems Testingcited by 8
- 2021Single and Multi-objective Test Cases Prioritization for Self-driving Cars in Virtual Environmentscited by 6
10 more
- 2022A Search-Based Framework for Automatic Generation of Testing Environments for Cyber-Physical Systemscited by 5
- 2023Towards Reliable AI: Adequacy Metrics for Ensuring the Quality of System-level Testing of Autonomous Vehiclescited by 5
- 2022Risk-Aware Scene Sampling for Dynamic Assurance of Autonomous Systemscited by 4
- 2022Machine Learning Testing in an ADAS Case Study Using Simulation-Integrated Bio-Inspired Search-Based Testingcited by 4
- 2023OpenSBT: A Modular Framework for Search-based Testing of Automated Driving Systemscited by 4
- 2023Reinforcement Learning Informed Evolutionary Search for Autonomous Systems Testingcited by 4
- 2023Test Generation Strategies for Building Failure Models and Explaining Spurious Failurescited by 4
- 2023Reflections on Surrogate-Assisted Search-Based Testing: A Taxonomy and Two Replication Studies based on Industrial ADAS and Simulink Modelscited by 3
- 2024Guiding the Search Towards Failure-Inducing Test Inputs Using Support Vector Machinescited by 2
- 2017Searching for test data with feature diversitycited by 1
Built on by
Citing papers per year: 2025: 1.
1 paper.