mlsanityData to 14 Oct 2026

Machine Learning Testing: Survey, Landscapes and Horizons

arXiv 1906.10742 ↗First posted 19 Jun 2019cs.LG cs.AI cs.SEcited by 218

Not on the map. Why?

Abstract (arXiv)

This paper provides a comprehensive survey of Machine Learning Testing (ML testing) research. It covers 144 papers on testing properties (e.g., correctness, robustness, and fairness), testing components (e.g., the data, learning program, and framework), testing workflow (e.g., test generation and test evaluation), and application scenarios (e.g., autonomous driving, machine translation). The paper also analyses trends concerning datasets, research trends, and research focus, concluding with research challenges and promising research directions in ML testing.

Key facts. Cited by 218 papers in the map (data through 14 Oct 2026). First posted on arXiv 19 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 41; 2022: 51; 2023: 42; 2024: 41; 2025: 26; 2026: 17.

218 papers.

Sort:

By area:

  1. 2021Trustworthy AI: From Principles to Practicescited by 207
  2. 2024Large Language Model-Based Agents for Software Engineering: A Surveycited by 178
  3. 2022Dawn of the transformer era in speech emotion recognition: closing the valence gapcited by 157
  4. 2022Bias Mitigation for Machine Learning Classifiers: A Comprehensive Surveycited by 125
  5. 2021Leveraging Automated Unit Tests for Unsupervised Code Translationcited by 82
  6. 2021Software Engineering for AI-Based Systems: A Surveycited by 69
  7. 2023Emotionally Numb or Empathetic? Evaluating How LLMs Feel Using EmotionBenchcited by 54
  8. 2023BiasAsker: Measuring the Bias in Conversational AI Systemcited by 53
  9. 2022Evaluation Gaps in Machine Learning Practicecited by 49
  10. 2023Stealthy Backdoor Attack for Code Modelscited by 49
  11. 2023Evaluating Superhuman Models with Consistency Checkscited by 47
  12. 2021How to Certify Machine Learning Based Safety-critical Systems? A Systematic Literature Reviewcited by 39
38 more
  1. 2021Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safetycited by 35
  2. 2021ModelDiff: Testing-Based DNN Similarity Comparison for Model Reuse Detectioncited by 33
  3. 2022A Comprehensive Empirical Study of Bias Mitigation Methods for Machine Learning Classifierscited by 30
  4. 2021DeepHyperion: Exploring the Feature Space of Deep Learning-Based Systems through Illumination Searchcited by 29
  5. 2024LLM Comparator: Visual Analytics for Side-by-Side Evaluation of Large Language Modelscited by 29
  6. 2022Fairness-aware Configuration of Machine Learning Librariescited by 28
  7. 2024Inherent Trade-Offs between Diversity and Stability in Multi-Task Benchmarkscited by 28
  8. 2021Taxonomy of Machine Learning Safety: A Survey and Primercited by 25
  9. 2021Silent Bugs in Deep Learning Frameworks: An Empirical Study of Keras and TensorFlowcited by 25
  10. 2022BAFFLE: Hiding Backdoors in Offline Reinforcement Learning Datasetscited by 25
  11. 2021Can Offline Testing of Deep Neural Networks Replace Their Online Testing?cited by 24
  12. 2021A Survey on Machine Learning Techniques for Source Code Analysiscited by 24
  13. 2025Hallucination Detection in Large Language Models with Metamorphic Relationscited by 24
  14. 2021Distribution-Aware Testing of Neural Networks Using Generative Modelscited by 22
  15. 2023MTTM: Metamorphic Testing for Textual Content Moderation Softwarecited by 22
  16. 2023A Meta-Summary of Challenges in Building Products with ML Components -- Collecting Experiences from 4758+ Practitionerscited by 22
  17. 2021An Empirical Study on Deployment Faults of Deep Learning Based Mobile Applicationscited by 20
  18. 2021Memorization and Generalization in Neural Code Intelligence Modelscited by 19
  19. 2022Many-Objective Reinforcement Learning for Online Testing of DNN-Enabled Systemscited by 19
  20. 2023To Be Forgotten or To Be Fair: Unveiling Fairness Implications of Machine Unlearning Methodscited by 19
  21. 2022Symphony: Composing Interactive Interfaces for Machine Learningcited by 18
  22. 2022NPC: Neuron Path Coverage via Characterizing Decision Logic of Deep Neural Networkscited by 18
  23. 2022When and Why Test Generators for Deep Learning Produce Invalid Inputs: an Empirical Studycited by 18
  24. 2022DeepFD: Automated Fault Diagnosis and Localization for Deep Learning Programscited by 16
  25. 2024New Job, New Gender? Measuring the Social Bias in Image Generation Modelscited by 16
  26. 2022AEON: A Method for Automatic Evaluation of NLP Test Casescited by 15
  27. 2022Bugs in Machine Learning-based Systems: A Faultload Benchmarkcited by 15
  28. 2024Robust Neural Information Retrieval: An Adversarial and Out-of-distribution Perspectivecited by 15
  29. 2022Ergo, SMIRK is Safe: A Safety Case for a Machine Learning Component in a Pedestrian Automatic Emergency Brake Systemcited by 14
  30. 2023Towards Fair Machine Learning Software: Understanding and Addressing Model Bias Through Counterfactual Thinkingcited by 14
  31. 2023Evaluating the Robustness of Interpretability Methods through Explanation Invariance and Equivariancecited by 14
  32. 2022Trust in AI: Interpretability is not necessary or sufficient, while black-box interaction is necessary and sufficientcited by 13
  33. 2023Testing of Deep Reinforcement Learning Agents with Surrogate Modelscited by 13
  34. 2021DeepMetis: Augmenting a Deep Learning Test Set to Increase its Mutation Scorecited by 12
  35. 2021Understanding Performance Problems in Deep Learning Systemscited by 12
  36. 2021MDPFuzz: Testing Models Solving Markov Decision Processescited by 12
  37. 2022QVIP: An ILP-based Formal Verification Approach for Quantized Neural Networkscited by 12
  38. 2023POLAR-Express: Efficient and Precise Formal Reachability Analysis of Neural-Network Controlled Systemscited by 12

50 most cited of 218 All 218 (JSON)