# When and Why Test Generators for Deep Learning Produce Invalid Inputs: an Empirical Study

arXiv: https://arxiv.org/abs/2212.11368 · HTML: https://mlsanity.com/paper/2212.11368/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 61 references that are in the map.
- First posted on arXiv 21 Dec 2022.

## Builds on (most cited first, 26 of 26)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [Striving for Simplicity: The All Convolutional Net](https://mlsanity.com/paper/1412.6806/) (2014)
- [Generalized Out-of-Distribution Detection: A Survey](https://mlsanity.com/paper/2110.11334/) (2021)
- [DeepXplore: Automated Whitebox Testing of Deep Learning Systems](https://mlsanity.com/paper/1705.06640/) (2017)
- [DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Cars](https://mlsanity.com/paper/1708.08559/) (2017)
- [Machine Learning Testing: Survey, Landscapes and Horizons](https://mlsanity.com/paper/1906.10742/) (2019)
- [DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems](https://mlsanity.com/paper/1803.07519/) (2018)
- [Guiding Deep Learning System Testing using Surprise Adequacy](https://mlsanity.com/paper/1808.08444/) (2018)
- [Taxonomy of Real Faults in Deep Learning Systems](https://mlsanity.com/paper/1910.11015/) (2019)
- [DLFuzz: Differential Fuzzing Testing of Deep Learning Systems](https://mlsanity.com/paper/1808.09413/) (2018)
- [Misbehaviour Prediction for Autonomous Driving Systems](https://mlsanity.com/paper/1910.04443/) (2019)
- [Model-based Exploration of the Frontier of Behaviours for Deep Learning System Testing](https://mlsanity.com/paper/2007.02787/) (2020)
- [DeepHyperion: Exploring the Feature Space of Deep Learning-Based Systems through Illumination Search](https://mlsanity.com/paper/2107.06997/) (2021)
- [Towards Characterizing Adversarial Defects of Deep Learning Software from the Lens of Uncertainty](https://mlsanity.com/paper/2004.11573/) (2020)
- [Reducing DNN Labelling Cost using Surprise Adequacy: An Industrial Case Study for Autonomous Driving](https://mlsanity.com/paper/2006.00894/) (2020)
- [Distribution-Aware Testing of Neural Networks Using Generative Models](https://mlsanity.com/paper/2102.13602/) (2021)
- [SINVAD: Search-based Image Space Navigation for DNN Image Classifier Test Input Generation](https://mlsanity.com/paper/2005.09296/) (2020)
- [DeepMetis: Augmenting a Deep Learning Test Set to Increase its Mutation Score](https://mlsanity.com/paper/2109.07514/) (2021)
- [Exposing Previously Undetectable Faults in Deep Neural Networks](https://mlsanity.com/paper/2106.00576/) (2021)
- [Black-box Safety Analysis and Retraining of DNNs based on Feature Extraction and Clustering](https://mlsanity.com/paper/2201.05077/) (2022)
- [DeepSmartFuzzer: Reward Guided Test Generation For Deep Learning](https://mlsanity.com/paper/1911.10621/) (2019)

## Built on by (most cited first, 18 of 18)

- [Online Safety Analysis for LLMs: a Benchmark, an Assessment, and a Path Forward](https://mlsanity.com/paper/2404.08517/) (2024)
- [Evaluating the Robustness of Test Selection Methods for Deep Neural Networks](https://mlsanity.com/paper/2308.01314/) (2023)
- [LUNA: A Model-Based Universal Analysis Framework for Large Language Models](https://mlsanity.com/paper/2310.14211/) (2023)
- [Evaluating the Impact of Flaky Simulators on Testing Autonomous Driving Systems](https://mlsanity.com/paper/2311.18768/) (2023)
- [Bridging the Gap between Real-world and Synthetic Images for Testing Autonomous Driving Systems](https://mlsanity.com/paper/2408.13950/) (2024)
- [Benchmarking Generative AI Models for Deep Learning Test Input Generation](https://mlsanity.com/paper/2412.17652/) (2024)
- [Targeted Deep Learning System Boundary Testing](https://mlsanity.com/paper/2408.06258/) (2024)
- [GIST: Generated Inputs Sets Transferability in Deep Learning](https://mlsanity.com/paper/2311.00801/) (2023)
- [Can Search-Based Testing with Pareto Optimization Effectively Cover Failure-Revealing Test Inputs?](https://mlsanity.com/paper/2410.11769/) (2024)
- [Simulator Ensembles for Trustworthy Autonomous Driving Systems Testing](https://mlsanity.com/paper/2503.08936/) (2025)
- [TestifAI: Tomography-Based Testing for Deep Learning Systems](https://mlsanity.com/paper/2608.18900/) (2026)
- [Adopting Two Supervisors for Efficient Use of Large-Scale Remote Deep Neural Networks](https://mlsanity.com/paper/2304.02654/) (2023)
- [Generating Minimalist Adversarial Perturbations to Test Object-Detection Models: An Adaptive Multi-Metric Evolutionary Search Approach](https://mlsanity.com/paper/2404.17020/) (2024)
- [DiffGAN: A Test Generation Approach for Differential Testing of Deep Neural Networks for Image Analysis](https://mlsanity.com/paper/2410.19794/) (2024)
- [Generating Realistic, Diverse, and Fault-Revealing Inputs with Latent Space Interpolation for Testing Deep Neural Networks](https://mlsanity.com/paper/2503.17630/) (2025)
- [Towards Assessing Deep Learning Test Input Generators](https://mlsanity.com/paper/2504.02329/) (2025)
- [Effort-Optimized, Accuracy-Driven Labelling and Validation of Test Inputs for DL Systems: A Mixed-Integer Linear Programming Approach](https://mlsanity.com/paper/2507.04990/) (2025)
- [HyperNet-Adaptation for Diffusion-Based Test Case Generation](https://mlsanity.com/paper/2601.15041/) (2026)

## Abstract (arXiv)

Testing Deep Learning (DL) based systems inherently requires large and representative test sets to evaluate whether DL systems generalise beyond their training datasets. Diverse Test Input Generators (TIGs) have been proposed to produce artificial inputs that expose issues of the DL systems by triggering misbehaviours. Unfortunately, such generated inputs may be invalid, i.e., not recognisable as part of the input domain, thus providing an unreliable quality assessment. Automated validators can ease the burden of manually checking the validity of inputs for human testers, although input validity is a concept difficult to formalise and, thus, automate. In this paper, we investigate to what extent TIGs can generate valid inputs, according to both automated and human validators. We conduct a large empirical study, involving 2 different automated validators, 220 human assessors, 5 different TIGs and 3 classification tasks. Our results show that 84% artificially generated inputs are valid, according to automated validators, but their expected label is not always preserved. Automated validators reach a good consensus with humans (78% accuracy), but still have limitations when dealing with feature-rich datasets.