# Bug Characterization in Machine Learning-based Systems arXiv: https://arxiv.org/abs/2307.14512 ยท HTML: https://mlsanity.com/paper/2307.14512/ ## Key facts - Cited by 10 papers in the map (data through 14 Oct 2026). - Builds on 21 of its 107 references that are in the map. - First posted on arXiv 26 Jul 2023. ## Builds on (most cited first, 21 of 21) - [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019) - [Caffe: Convolutional Architecture for Fast Feature Embedding](https://mlsanity.com/paper/1408.5093/) (2014) - [Machine Learning Testing: Survey, Landscapes and Horizons](https://mlsanity.com/paper/1906.10742/) (2019) - [MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems](https://mlsanity.com/paper/1512.01274/) (2015) - [Taxonomy of Real Faults in Deep Learning Systems](https://mlsanity.com/paper/1910.11015/) (2019) - [Software Engineering for AI-Based Systems: A Survey](https://mlsanity.com/paper/2105.01984/) (2021) - [A Comprehensive Study on Deep Learning Bug Characteristics](https://mlsanity.com/paper/1906.01388/) (2019) - [Model Complexity of Deep Learning: A Survey](https://mlsanity.com/paper/2103.05127/) (2021) - [DeepLocalize: Fault Localization for Deep Neural Networks](https://mlsanity.com/paper/2103.03376/) (2021) - [Repairing Deep Neural Networks: Fix Patterns and Challenges](https://mlsanity.com/paper/2005.00972/) (2020) - [Silent Bugs in Deep Learning Frameworks: An Empirical Study of Keras and TensorFlow](https://mlsanity.com/paper/2112.13314/) (2021) - [DeepDiagnosis: Automatically Diagnosing Faults and Recommending Actionable Fixes in Deep Learning Programs](https://mlsanity.com/paper/2112.04036/) (2021) - [Faults in Deep Reinforcement Learning Programs: A Taxonomy and A Detection Approach](https://mlsanity.com/paper/2101.00135/) (2021) - [Bugs in Machine Learning-based Systems: A Faultload Benchmark](https://mlsanity.com/paper/2206.12311/) (2022) - [Understanding Performance Problems in Deep Learning Systems](https://mlsanity.com/paper/2112.01771/) (2021) - [Automatic Fault Detection for Deep Learning Programs Using Graph Transformations](https://mlsanity.com/paper/2105.08095/) (2021) - [On Reporting Performance and Accuracy Bugs for Deep Learning Frameworks: An Exploratory Study from GitHub](https://mlsanity.com/paper/2204.07893/) (2022) - [The challenge of reproducible ML: an empirical study on the impact of bugs](https://mlsanity.com/paper/2109.03991/) (2021) - [What is the Connection Between Issues, Bugs, and Enhancements? (Lessons Learned from 800+ Software Projects)](https://mlsanity.com/paper/1710.08736/) (2017) - [Same File, Different Changes: The Potential of Meta-Maintenance on GitHub](https://mlsanity.com/paper/2102.06355/) (2021) - [Revisiting reopened bugs in open source software systems](https://mlsanity.com/paper/2202.08701/) (2022) ## Built on by (most cited first, 10 of 10) - [Bugs in Large Language Models Generated Code: An Empirical Study](https://mlsanity.com/paper/2403.08937/) (2024) - [Characterizing Faults in Agentic AI: A Taxonomy of Types, Symptoms, and Root Causes](https://mlsanity.com/paper/2603.06847/) (2026) - [Software Engineering and Foundation Models: Insights from Industry Blogs Using a Jury of Foundation Models](https://mlsanity.com/paper/2410.09012/) (2024) - [Real Faults in Model Context Protocol (MCP) Software: a Comprehensive Taxonomy](https://mlsanity.com/paper/2603.05637/) (2026) - [PeaTMOSS: Mining Pre-Trained Models in Open-Source Software](https://mlsanity.com/paper/2310.03620/) (2023) - [Why Attention Fails: A Taxonomy of Faults in Attention-Based Neural Networks](https://mlsanity.com/paper/2508.04925/) (2025) - [Imitation Game: Reproducing Deep Learning Bugs Leveraging an Intelligent Agent](https://mlsanity.com/paper/2512.14990/) (2025) - [DepRadar: Agentic Coordination for Context Aware Defect Impact Analysis in Deep Learning Libraries](https://mlsanity.com/paper/2601.09440/) (2026) - [Data-aware Static Analysis: Improving Detection of Semantic Faults in Machine Learning Code Using Data Characteristics](https://mlsanity.com/paper/2606.09957/) (2026) - [How Do AI Coding Agents Contribute to Software Development? an Empirical Study of Agentic Pull Requests](https://mlsanity.com/paper/2607.21832/) (2026) ## Abstract (arXiv) Rapid growth of applying Machine Learning (ML) in different domains, especially in safety-critical areas, increases the need for reliable ML components, i.e., a software component operating based on ML. Understanding the bugs characteristics and maintenance challenges in ML-based systems can help developers of these systems to identify where to focus maintenance and testing efforts, by giving insights into the most error-prone components, most common bugs, etc. In this paper, we investigate the characteristics of bugs in ML-based software systems and the difference between ML and non-ML bugs from the maintenance viewpoint. We extracted 447,948 GitHub repositories that used one of the three most popular ML frameworks, i.e., TensorFlow, Keras, and PyTorch. After multiple filtering steps, we select the top 300 repositories with the highest number of closed issues. We manually investigate the extracted repositories to exclude non-ML-based systems. Our investigation involved a manual inspection of 386 sampled reported issues in the identified ML-based systems to indicate whether they affect ML components or not. Our analysis shows that nearly half of the real issues reported in ML-based systems are ML bugs, indicating that ML components are more error-prone than non-ML components. Next, we thoroughly examined 109 identified ML bugs to identify their root causes, symptoms, and calculate their required fixing time. The results also revealed that ML bugs have significantly different characteristics compared to non-ML bugs, in terms of the complexity of bug-fixing (number of commits, changed files, and changed lines of code). Based on our results, fixing ML bugs are more costly and ML components are more error-prone, compared to non-ML bugs and non-ML components respectively. Hence, paying a significant attention to the reliability of the ML components is crucial in ML-based systems.