# Black-Box Testing of Deep Neural Networks Through Test Case Diversity

arXiv: https://arxiv.org/abs/2112.12591 · HTML: https://mlsanity.com/paper/2112.12591/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 95 references that are in the map.
- First posted on arXiv 20 Dec 2021.

## Builds on (most cited first, 22 of 22)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction](https://mlsanity.com/paper/1802.03426/) (2018)
- [DeepXplore: Automated Whitebox Testing of Deep Learning Systems](https://mlsanity.com/paper/1705.06640/) (2017)
- [DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars](https://mlsanity.com/paper/1708.08559/) (2017)
- [Deep Reinforcement Learning framework for Autonomous Driving](https://mlsanity.com/paper/1704.02532/) (2017)
- [DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems](https://mlsanity.com/paper/1803.07519/) (2018)
- [Guiding Deep Learning System Testing using Surprise Adequacy](https://mlsanity.com/paper/1808.08444/) (2018)
- [Misbehaviour Prediction for Autonomous Driving Systems](https://mlsanity.com/paper/1910.04443/) (2019)
- [Diversity in Machine Learning](https://mlsanity.com/paper/1807.01477/) (2018)
- [Fair and Diverse DPP-based Data Summarization](https://mlsanity.com/paper/1802.04023/) (2018)
- [Mean Deviation Similarity Index: Efficient and Reliable Full-Reference Image Quality Evaluator](https://mlsanity.com/paper/1608.07433/) (2016)
- [DeepHyperion: Exploring the Feature Space of Deep Learning-Based Systems through Illumination Search](https://mlsanity.com/paper/2107.06997/) (2021)
- [GDPP: Learning Diverse Generations Using Determinantal Point Process](https://mlsanity.com/paper/1812.00068/) (2018)
- [Importance-Driven Deep Learning System Testing](https://mlsanity.com/paper/2002.03433/) (2020)
- [Boosting Operational DNN Testing Efficiency through Conditioning](https://mlsanity.com/paper/1906.02533/) (2019)
- [An Empirical Study of Fault Localization Families and Their Combinations](https://mlsanity.com/paper/1803.09939/) (2018)
- [Fast Image Scanning with Deep Max-Pooling Convolutional Neural Networks](https://mlsanity.com/paper/1302.1700/) (2013)
- [Supporting DNN Safety Analysis and Retraining through Heatmap-based Unsupervised Learning](https://mlsanity.com/paper/2002.00863/) (2020)
- [Test Set Diameter: Quantifying the Diversity of Sets of Test Cases](https://mlsanity.com/paper/1506.03482/) (2015)
- [There is Limited Correlation between Coverage and Robustness for Deep Neural Networks](https://mlsanity.com/paper/1911.05904/) (2019)
- [Towards Improved Testing For Deep Learning](https://mlsanity.com/paper/1902.06320/) (2019)
- [On Normalized Compression Distance and Large Malware](https://mlsanity.com/paper/1509.00689/) (2015)

## Built on by (most cited first, 18 of 18)

- [A Search-Based Testing Approach for Deep Reinforcement Learning Agents](https://mlsanity.com/paper/2206.07813/) (2022)
- [DeepGD: A Multi-Objective Black-Box Test Selection Approach for Deep Neural Networks](https://mlsanity.com/paper/2303.04878/) (2023)
- [SMARLA: A Safety Monitoring Approach for Deep Reinforcement Learning Agents](https://mlsanity.com/paper/2308.02594/) (2023)
- [Pitfalls in Language Models for Code Intelligence: A Taxonomy and Survey](https://mlsanity.com/paper/2310.17903/) (2023)
- [MetaSel: A Test Selection Approach for Fine-tuned DNN Models](https://mlsanity.com/paper/2503.17534/) (2025)
- [The Pursuit of Diversity: Multi-Objective Testing of Deep Reinforcement Learning Agents](https://mlsanity.com/paper/2510.14727/) (2025)
- [GIST: Generated Inputs Sets Transferability in Deep Learning](https://mlsanity.com/paper/2311.00801/) (2023)
- [Can Search-Based Testing with Pareto Optimization Effectively Cover Failure-Revealing Test Inputs?](https://mlsanity.com/paper/2410.11769/) (2024)
- [A Highly Efficient Diversity-based Input Selection for DNN Improvement Using VLMs](https://mlsanity.com/paper/2601.08024/) (2026)
- [Effective Black Box Testing of Sentiment Analysis Classification Networks](https://mlsanity.com/paper/2407.20884/) (2024)
- [Robust Black-box Testing of Deep Neural Networks using Co-Domain Coverage](https://mlsanity.com/paper/2408.06766/) (2024)
- [LeCov: Multi-level Testing Criteria for Large Language Models](https://mlsanity.com/paper/2408.10474/) (2024)
- [DiffGAN: A Test Generation Approach for Differential Testing of Deep Neural Networks for Image Analysis](https://mlsanity.com/paper/2410.19794/) (2024)
- [Towards Understanding Deep Learning Model in Image Recognition via Coverage Test](https://mlsanity.com/paper/2505.08814/) (2025)
- [TopoMap: A Feature-based Semantic Discriminator of the Topographical Regions in the Test Input Space](https://mlsanity.com/paper/2509.03242/) (2025)
- [Fuzzing the brain: Automated stress testing for the safety of ML-driven neurostimulation](https://mlsanity.com/paper/2512.05383/) (2025)
- [Towards Reliable Testing of Machine Unlearning](https://mlsanity.com/paper/2604.16536/) (2026)
- [CAFD: Concept-Aware DNN Fault Detection using VLMs](https://mlsanity.com/paper/2605.24008/) (2026)

## Abstract (arXiv)

Deep Neural Networks (DNNs) have been extensively used in many areas including image processing, medical diagnostics, and autonomous driving. However, DNNs can exhibit erroneous behaviours that may lead to critical errors, especially when used in safety-critical systems. Inspired by testing techniques for traditional software systems, researchers have proposed neuron coverage criteria, as an analogy to source code coverage, to guide the testing of DNN models. Despite very active research on DNN coverage, several recent studies have questioned the usefulness of such criteria in guiding DNN testing. Further, from a practical standpoint, these criteria are white-box as they require access to the internals or training data of DNN models, which is in many contexts not feasible or convenient. In this paper, we investigate black-box input diversity metrics as an alternative to white-box coverage criteria. To this end, we first select and adapt three diversity metrics and study, in a controlled manner, their capacity to measure actual diversity in input sets. We then analyse their statistical association with fault detection using four datasets and five DNN models. We further compare diversity with state-of-the-art white-box coverage criteria. Our experiments show that relying on the diversity of image features embedded in test input sets is a more reliable indicator than coverage criteria to effectively guide the testing of DNNs. Indeed, we found that one of our selected black-box diversity metrics far outperforms existing coverage criteria in terms of fault-revealing capability and computational time. Results also confirm the suspicions that state-of-the-art coverage metrics are not adequate to guide the construction of test input sets to detect as many faults as possible with natural inputs.