# DeepGD: A Multi-Objective Black-Box Test Selection Approach for Deep Neural Networks

arXiv: https://arxiv.org/abs/2303.04878 · HTML: https://mlsanity.com/paper/2303.04878/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 76 references that are in the map.
- First posted on arXiv 8 Mar 2023.

## Builds on (most cited first, 28 of 28)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Survey on Deep Learning in Medical Image Analysis](https://mlsanity.com/paper/1702.05747/) (2017)
- [Deep Batch Active Learning by Diverse, Uncertain Gradient Lower Bounds](https://mlsanity.com/paper/1906.03671/) (2019)
- [A Survey of Deep Learning Techniques for Autonomous Driving](https://mlsanity.com/paper/1910.07738/) (2019)
- [DeepXplore: Automated Whitebox Testing of Deep Learning Systems](https://mlsanity.com/paper/1705.06640/) (2017)
- [DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars](https://mlsanity.com/paper/1708.08559/) (2017)
- [pymoo: Multi-objective Optimization in Python](https://mlsanity.com/paper/2002.04504/) (2020)
- [DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems](https://mlsanity.com/paper/1803.07519/) (2018)
- [Guiding Deep Learning System Testing using Surprise Adequacy](https://mlsanity.com/paper/1808.08444/) (2018)
- [Deep Active Learning: Unified and Principled Method for Query and Training](https://mlsanity.com/paper/1911.09162/) (2019)
- [A Comparative Survey of Deep Active Learning](https://mlsanity.com/paper/2203.13450/) (2022)
- [Diversity in Machine Learning](https://mlsanity.com/paper/1807.01477/) (2018)
- [How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Review](https://mlsanity.com/paper/2107.12045/) (2021)
- [Survey on Deep Neural Networks in Speech and Vision Systems](https://mlsanity.com/paper/1908.07656/) (2019)
- [Testing Deep Neural Networks](https://mlsanity.com/paper/1803.04792/) (2018)
- [DeepHyperion: Exploring the Feature Space of Deep Learning-Based Systems through Illumination Search](https://mlsanity.com/paper/2107.06997/) (2021)
- [Test Selection for Deep Learning Systems](https://mlsanity.com/paper/1904.13195/) (2019)
- [Simple Techniques Work Surprisingly Well for Neural Network Test Prioritization and Active Learning (Replicability Study)](https://mlsanity.com/paper/2205.00664/) (2022)
- [Importance-Driven Deep Learning System Testing](https://mlsanity.com/paper/2002.03433/) (2020)
- [Boosting Operational DNN Testing Efficiency through Conditioning](https://mlsanity.com/paper/1906.02533/) (2019)
- [Black-Box Testing of Deep Neural Networks Through Test Case Diversity](https://mlsanity.com/paper/2112.12591/) (2021)
- [Revisiting Neuron Coverage Metrics and Quality of Deep Neural Networks](https://mlsanity.com/paper/2201.00191/) (2022)
- [A Search-Based Testing Approach for Deep Reinforcement Learning Agents](https://mlsanity.com/paper/2206.07813/) (2022)
- [Test Case Selection and Prioritization Using Machine Learning: A Systematic Literature Review](https://mlsanity.com/paper/2106.13891/) (2021)
- [Supporting DNN Safety Analysis and Retraining through Heatmap-based Unsupervised Learning](https://mlsanity.com/paper/2002.00863/) (2020)
- [Test Set Diameter: Quantifying the Diversity of Sets of Test Cases](https://mlsanity.com/paper/1506.03482/) (2015)
- [Black-box Safety Analysis and Retraining of DNNs based on Feature Extraction and Clustering](https://mlsanity.com/paper/2201.05077/) (2022)
- [ATM: Black-box Test Case Minimization based on Test Code Similarity and Evolutionary Search](https://mlsanity.com/paper/2210.16269/) (2022)

## Built on by (most cited first, 11 of 11)

- [VLATest: Testing and Evaluating Vision-Language-Action Models for Robotic Manipulation](https://mlsanity.com/paper/2409.12894/) (2024)
- [SMARLA: A Safety Monitoring Approach for Deep Reinforcement Learning Agents](https://mlsanity.com/paper/2308.02594/) (2023)
- [MetaSel: A Test Selection Approach for Fine-tuned DNN Models](https://mlsanity.com/paper/2503.17534/) (2025)
- [A Highly Efficient Diversity-based Input Selection for DNN Improvement Using VLMs](https://mlsanity.com/paper/2601.08024/) (2026)
- [Robust Black-box Testing of Deep Neural Networks using Co-Domain Coverage](https://mlsanity.com/paper/2408.06766/) (2024)
- [DiffGAN: A Test Generation Approach for Differential Testing of Deep Neural Networks for Image Analysis](https://mlsanity.com/paper/2410.19794/) (2024)
- [TopoMap: A Feature-based Semantic Discriminator of the Topographical Regions in the Test Input Space](https://mlsanity.com/paper/2509.03242/) (2025)
- [Learning-Based Testing for Deep Learning: Enhancing Model Robustness with Adversarial Input Prioritization](https://mlsanity.com/paper/2509.23961/) (2025)
- [Bias Testing and Mitigation in Black Box LLMs using Metamorphic Relations](https://mlsanity.com/paper/2512.00556/) (2025)
- [CAFD: Concept-Aware DNN Fault Detection using VLMs](https://mlsanity.com/paper/2605.24008/) (2026)
- [Learnable Weighting of Intra-Attribute Distances for Categorical Data Clustering with Nominal and Ordinal Attributes](https://mlsanity.com/paper/2607.05464/) (2026)

## Abstract (arXiv)

Deep neural networks (DNNs) are widely used in various application domains such as image processing, speech recognition, and natural language processing. However, testing DNN models may be challenging due to the complexity and size of their input domain. Particularly, testing DNN models often requires generating or exploring large unlabeled datasets. In practice, DNN test oracles, which identify the correct outputs for inputs, often require expensive manual effort to label test data, possibly involving multiple experts to ensure labeling correctness. In this paper, we propose DeepGD, a black-box multi-objective test selection approach for DNN models. It reduces the cost of labeling by prioritizing the selection of test inputs with high fault revealing power from large unlabeled datasets. DeepGD not only selects test inputs with high uncertainty scores to trigger as many mispredicted inputs as possible but also maximizes the probability of revealing distinct faults in the DNN model by selecting diverse mispredicted inputs. The experimental results conducted on four widely used datasets and five DNN models show that in terms of fault-revealing ability: (1) White-box, coverage-based approaches fare poorly, (2) DeepGD outperforms existing black-box test selection approaches in terms of fault detection, and (3) DeepGD also leads to better guidance for DNN model retraining when using selected inputs to augment the training set.