mlsanityData to 7 Oct 2026

Exposing Previously Undetectable Faults in Deep Neural Networks

arXiv 2106.00576 ↗First posted 1 Jun 2021cs.LG cs.CV cs.SEcited by 11

Not on the map. Why?

Abstract (arXiv)

Existing methods for testing DNNs solve the oracle problem by constraining the raw features (e.g. image pixel values) to be within a small distance of a dataset example for which the desired DNN output is known. But this limits the kinds of faults these approaches are able to detect. In this paper, we introduce a novel DNN testing method that is able to find faults in DNNs that other methods cannot. The crux is that, by leveraging generative machine learning, we can generate fresh test inputs that vary in their high-level features (for images, these include object shape, location, texture, and colour). We demonstrate that our approach is capable of detecting deliberately injected faults as well as new faults in state-of-the-art DNNs, and that in both cases, existing methods are unable to find these faults.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 32 of its 50 references that are in the map. First posted on arXiv 1 Jun 2021.

Builds on

32 of 50 references.

References · 32

  1. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  2. 2013Auto-Encoding Variational Bayescited by 3,980
  3. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  4. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  5. 2014Conditional Generative Adversarial Netscited by 2,152
  6. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  7. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,430
  8. 2019Natural Adversarial Examplescited by 1,401
  9. 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 950
  10. 2020Fast is better than free: Revisiting adversarial trainingcited by 723
  11. 2017Unsupervised Image-to-Image Translation Networkscited by 602
  12. 2020Do Adversarially Robust ImageNet Models Transfer Better?cited by 315
20 more
  1. 2017DeepXplore: Automated Whitebox Testing of Deep Learning Systemscited by 307
  2. 2016NIPS 2016 Tutorial: Generative Adversarial Networkscited by 284
  3. 2017DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Carscited by 249
  4. 2019On the "steerability" of generative adversarial networkscited by 247
  5. 2017Generalization and Equilibrium in Generative Adversarial Nets (GANs)cited by 165
  6. 2019MNIST-C: A Robustness Benchmark for Computer Visioncited by 137
  7. 2018Constructing Unrestricted Adversarial Examples with Generative Modelscited by 135
  8. 2019SemanticAdv: Generating Adversarial Examples via Attribute-conditional Image Editingcited by 93
  9. 2018Semantic Adversarial Examplescited by 86
  10. 2019Towards Large yet Imperceptible Adversarial Image Perturbations with Perceptual Color Distancecited by 72
  11. 2018DLFuzz: Differential Fuzzing Testing of Deep Learning Systemscited by 64
  12. 2018Motivating the Rules of the Game for Adversarial Example Researchcited by 57
  13. 2019Semantic Adversarial Attacks: Parametric Transformations That Fool Deep Classifierscited by 56
  14. 2020Fundamental Tradeoffs between Invariance and Sensitivity to Adversarial Perturbationscited by 55
  15. 2020Safety Concerns and Mitigation Approaches Regarding the Use of Deep Learning in Safety-Critical Perception Taskscited by 31
  16. 2019Achieving Robustness in the Wild via Adversarial Mixing with Disentangled Representationscited by 28
  17. 2018Semantic Adversarial Deep Learningcited by 15
  18. 2020Generating Semantic Adversarial Examples via Feature Manipulationcited by 4
  19. 2019Testing Deep Learning Models for Image Analysis Using Object-Relevant Metamorphic Relationscited by 3
  20. 2019Analyzing and Improving Neural Networks by Generating Semantic Counterexamples through Differentiable Renderingcited by 2

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 4; 2023: 2; 2024: 3; 2025: 1.

11 papers.

  1. 2021Partial success in closing the gap between human and machine visioncited by 185
  2. 2022When and Why Test Generators for Deep Learning Produce Invalid Inputs: an Empirical Studycited by 18
  3. 2022Hierarchical Distribution-Aware Testing of Deep Learningcited by 8
  4. 2022Generating and Detecting True Ambiguity: A Forgotten Danger in DNN Supervision Testingcited by 8
  5. 2022Provably Tightest Linear Approximation for Robustness Verification of Sigmoid-like Neural Networkscited by 8
  6. 2023Precise and Generalized Robustness Certification for Neural Networkscited by 4
  7. 2024Benchmarking Generative AI Models for Deep Learning Test Input Generationcited by 4
  8. 2024Targeted Deep Learning System Boundary Testingcited by 3
  9. 2023GIST: Generated Inputs Sets Transferability in Deep Learningcited by 1
  10. 2024A3Rank: Augmentation Alignment Analysis for Prioritizing Overconfident Failing Samples for Deep Learning Modelscited by 0
  11. 2025Formal Reasoning About Confidence and Automated Verification of Neural Networkscited by 0