mlsanityData to 7 Oct 2026

Software Testing for Machine Learning

arXiv 2205.00210 ↗First posted 30 Apr 2022cs.SE cs.AI cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Machine learning has become prevalent across a wide variety of applications. Unfortunately, machine learning has also shown to be susceptible to deception, leading to errors, and even fatal failures. This circumstance calls into question the widespread use of machine learning, especially in safety-critical applications, unless we are able to assure its correctness and trustworthiness properties. Software verification and testing are established technique for assuring such properties, for example by detecting errors. However, software testing challenges for machine learning are vast and profuse - yet critical to address. This summary talk discusses the current state-of-the-art of software testing for machine learning. More specifically, it discusses six key challenge areas for software testing of machine learning systems, examines current approaches to these challenges and highlights their limitations. The paper provides a research agenda with elaborated directions for making progress toward advancing the state-of-the-art on testing of machine learning.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 34 of its 62 references that are in the map. First posted on arXiv 30 Apr 2022.

Builds on

34 of 62 references.

References · 34

  1. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  2. 2013Intriguing properties of neural networkscited by 5,114
  3. 2016Towards Evaluating the Robustness of Neural Networkscited by 2,976
  4. 2015DeepFool: a simple and accurate method to fool deep neural networkscited by 1,315
  5. 2017Ensemble Adversarial Training: Attacks and Defensescited by 833
  6. 2016Universal adversarial perturbationscited by 774
  7. 2017Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networkscited by 739
  8. 2015Distillation as a Defense to Adversarial Perturbations against Deep Neural Networkscited by 675
  9. 2017Adversarial Examples for Evaluating Reading Comprehension Systemscited by 568
  10. 2017Adversarial Examples: Attacks and Defenses for Deep Learningcited by 350
  11. 2017DeepXplore: Automated Whitebox Testing of Deep Learning Systemscited by 307
  12. 2017Adversarial Examples for Semantic Segmentation and Object Detectioncited by 275
22 more
  1. 2018Audio Adversarial Examples: Targeted Attacks on Speech-to-Textcited by 268
  2. 2016Safety Verification of Deep Neural Networkscited by 262
  3. 2018Generating Adversarial Examples with Adversarial Networkscited by 256
  4. 2017DeepTest: Automated Testing of Deep-Neural-Network-driven Autonomous Carscited by 249
  5. 2017Formal Guarantees on the Robustness of a Classifier against Adversarial Manipulationcited by 182
  6. 2018DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systemscited by 144
  7. 2018Constructing Unrestricted Adversarial Examples with Generative Modelscited by 135
  8. 2018Adversarial Attacks and Defences Competitioncited by 96
  9. 2018Reachability Analysis of Deep Neural Networks with Provable Guaranteescited by 93
  10. 2016Measuring Neural Net Robustness with Constraintscited by 90
  11. 2015Analysis of classifiers' robustness to adversarial perturbationscited by 76
  12. 2018Interpretable Adversarial Perturbation in Input Embedding Space for Textcited by 67
  13. 2018TensorFuzz: Debugging Neural Networks with Coverage-Guided Fuzzingcited by 67
  14. 2018DeepMutation: Mutation Testing of Deep Learning Systemscited by 66
  15. 2018Concolic Testing for Deep Neural Networkscited by 51
  16. 2017Feature-Guided Black-Box Safety Testing of Deep Neural Networkscited by 48
  17. 2018Testing Deep Neural Networkscited by 36
  18. 2018Identifying Implementation Bugs in Machine Learning based Image Classifiers using Metamorphic Testingcited by 30
  19. 2017Foolbox: A Python toolbox to benchmark the robustness of machine learning modelscited by 19
  20. 2018Combinatorial Testing for Deep Learning Systemscited by 8
  21. 2018DeepCruiser: Automated Guided Testing for Stateful Deep Learning Systemscited by 4
  22. 2019DeepGini: Prioritizing Massive Tests to Enhance the Robustness of Deep Neural Networkscited by 1

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 3; 2024: 1.

5 papers.

  1. 2023An Empirical Study of Pre-Trained Model Reuse in the Hugging Face Deep Learning Model Registrycited by 45
  2. 2023Inferring Capabilities from Task Performance with Bayesian Triangulationcited by 13
  3. 2022Differential testing for machine learning: an analysis for classification algorithms beyond deep learningcited by 3
  4. 2023ReMAV: Reward Modeling of Autonomous Vehicles for Finding Likely Failure Eventscited by 0
  5. 2024LeCov: Multi-level Testing Criteria for Large Language Modelscited by 0