mlsanityData to 7 Oct 2026

A Comprehensive Study on Deep Learning Bug Characteristics

arXiv 1906.01388 ↗First posted 3 Jun 2019cs.SE cs.LGcited by 65

Not on the map. Why?

Abstract (arXiv)

Deep learning has gained substantial popularity in recent years. Developers mainly rely on libraries and tools to add deep learning capabilities to their software. What kinds of bugs are frequently found in such software? What are the root causes of such bugs? What impacts do such bugs have? Which stages of deep learning pipeline are more bug prone? Are there any antipatterns? Understanding such characteristics of bugs in deep learning software has the potential to foster the development of better deep learning platforms, debugging mechanisms, development practices, and encourage the development of analysis and verification frameworks. Therefore, we study 2716 high-quality posts from Stack Overflow and 500 bug fix commits from Github about five popular deep learning libraries Caffe, Keras, Tensorflow, Theano, and Torch to understand the types of bugs, root causes of bugs, impacts of bugs, bug-prone stage of deep learning pipeline as well as whether there are some common antipatterns found in this buggy software. The key findings of our study include: data bug and logic bug are the most severe bug types in deep learning software appearing more than 48% of the times, major root causes of these bugs are Incorrect Model Parameter (IPS) and Structural Inefficiency (SI) showing up more than 43% of the times. We have also found that the bugs in the usage of deep learning libraries have some common antipatterns that lead to a strong correlation of bug types among the libraries.

Key facts. Cited by 65 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 12; 2022: 11; 2023: 11; 2024: 12; 2025: 8; 2026: 11.

65 papers.

Sort:

By area:

  1. 2021Software Engineering for AI-Based Systems: A Surveycited by 68
  2. 2024What's Wrong with Your Code Generated by Large Language Models? An Extensive Studycited by 52
  3. 2021Fair Preprocessing: Towards Understanding Compositional Fairness of Data Transformers in Machine Learning Pipelinecited by 49
  4. 2021Silent Bugs in Deep Learning Frameworks: An Empirical Study of Keras and TensorFlowcited by 25
  5. 2021An Empirical Study on Deployment Faults of Deep Learning Based Mobile Applicationscited by 20
  6. 2021DeepDiagnosis: Automatically Diagnosing Faults and Recommending Actionable Fixes in Deep Learning Programscited by 19
  7. 2021Faults in Deep Reinforcement Learning Programs: A Taxonomy and A Detection Approachcited by 18
  8. 2022DeepFD: Automated Fault Diagnosis and Localization for Deep Learning Programscited by 16
  9. 2022Bugs in Machine Learning-based Systems: A Faultload Benchmarkcited by 15
  10. 2021DeepMetis: Augmenting a Deep Learning Test Set to Increase its Mutation Scorecited by 12
  11. 2021Understanding Performance Problems in Deep Learning Systemscited by 12
  12. 2023Challenges and Practices of Deep Learning Model Reengineering: A Case Study on Computer Visioncited by 11
38 more
  1. 2021Design Smells in Deep Learning Programs: An Empirical Studycited by 10
  2. 2023Bug Characterization in Machine Learning-based Systemscited by 10
  3. 2021Automatic Fault Detection for Deep Learning Programs Using Graph Transformationscited by 9
  4. 2022Code Smells for Machine Learning Applicationscited by 7
  5. 2024A Survey on Failure Analysis and Fault Injection in AI Systemscited by 7
  6. 2024Understanding the Fundamental Design Decisions of Retrieval-Augmented Generation Systemscited by 7
  7. 2024Towards Enhancing the Reproducibility of Deep Learning Bugs: An Empirical Studycited by 6
  8. 2026When Agents Fail: A Comprehensive Study of Bugs in LLM Agents with Automated Labelingcited by 6
  9. 2021Yes We Care! -- Certification for Machine Learning Methods through the Care Label Frameworkcited by 5
  10. 2021Manas: Mining Software Repositories to Assist AutoMLcited by 5
  11. 2023Mutation-based Fault Localization of Deep Neural Networkscited by 5
  12. 2022Data Smells: Categories, Causes and Consequences, and Detection of Suspicious Data in AI-based Systemscited by 4
  13. 2022DNNAbacus: Toward Accurate Computational Cost Prediction for Deep Neural Networkscited by 4
  14. 2023An investigation of challenges encountered when specifying training data and runtime monitors for safety critical ML applicationscited by 4
  15. 2024Deep Learning Library Testing: Definition, Methods and Challengescited by 4
  16. 2024RogueGPT: dis-ethical tuning transforms ChatGPT4 into a Rogue AI in 158 Wordscited by 4
  17. 2025Deep-Bench: Deep Learning Benchmark Dataset for Code Generationcited by 3
  18. 2026Will It Survive? Deciphering the Fate of AI-Generated Code in Open Sourcecited by 3
  19. 2026CR-Bench: Evaluating the Real-World Utility of AI Code Review Agentscited by 3
  20. 2026Agent Skills Can Be Harmful: An Empirical Study of Skill-Induced Failures in LLM Agentscited by 3
  21. 2022Challenges in Migrating Imperative Deep Learning Programs to Graph Execution: An Empirical Studycited by 2
  22. 2022What is Software Quality for AI Engineers? Towards a Thinning of the Fogcited by 2
  23. 2022Testing Feedforward Neural Networks Training Programscited by 2
  24. 2022Software Engineering Approaches for TinyML based IoT Embedded Vision: A Systematic Literature Reviewcited by 2
  25. 2022An Empirical Study of Library Usage and Dependency in Deep Learning Frameworkscited by 2
  26. 2023What Kinds of Contracts Do ML APIs Need?cited by 2
  27. 2024Unraveling Code Clone Dynamics in Deep Learning Frameworkscited by 2
  28. 2026Why Does the LLM Stop Computing: An Empirical Study of User-Reported Failures in Open-Source LLMscited by 2
  29. 2023An Empirical Study on Bugs Inside PyTorch: A Replication Studycited by 1
  30. 2024Inferring Data Preconditions from Deep Learning Models for Trustworthy Prediction in Deploymentcited by 1
  31. 2024Toward Debugging Deep Reinforcement Learning Programs with RLExplorercited by 1
  32. 2025Why Attention Fails: A Taxonomy of Faults in Attention-Based Neural Networkscited by 1
  33. 2025xMem: A CPU-Based Approach for Accurate Estimation of GPU Memory in Deep Learning Training Workloadscited by 1
  34. 2025Imitation Game: Reproducing Deep Learning Bugs Leveraging an Intelligent Agentcited by 1
  35. 2026Using Small Language Models to Reverse-Engineer Machine Learning Pipelines Structurescited by 1
  36. 2026SelfHeal: Empirical Fix Pattern Analysis and Bug Repair in LLM Agentscited by 1
  37. 2026Understanding Agent-Reactive Bugs at the Model-Harness Boundary: An Empirical Study of LLM Agent Issue Reportscited by 1
  38. 2022Comparative analysis of real bugs in open-source Machine Learning projects -- A Registered Reportcited by 0

50 most cited of 65 All 65 (JSON)