# Toward Understanding Deep Learning Framework Bugs

arXiv: https://arxiv.org/abs/2203.04026 · HTML: https://mlsanity.com/paper/2203.04026/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Mar 2022.

## Built on by (most cited first, 26 of 26)

- [Impacts of floating-point non-associativity on reproducibility for HPC and deep learning applications](https://mlsanity.com/paper/2408.05148/) (2024)
- [Challenges and Practices of Deep Learning Model Reengineering: A Case Study on Computer Vision](https://mlsanity.com/paper/2303.07476/) (2023)
- [A Survey on Failure Analysis and Fault Injection in AI Systems](https://mlsanity.com/paper/2407.00125/) (2024)
- [Training with Confidence: Catching Silent Errors in Deep Learning Training with Automated Proactive Checks](https://mlsanity.com/paper/2506.14813/) (2025)
- [Understanding the Fundamental Design Decisions of Retrieval-Augmented Generation Systems](https://mlsanity.com/paper/2411.19463/) (2024)
- [Deep Learning Library Testing: Definition, Methods and Challenges](https://mlsanity.com/paper/2404.17871/) (2024)
- [Enhancing Differential Testing With LLMs For Testing Deep Learning Libraries](https://mlsanity.com/paper/2406.07944/) (2024)
- [COMET: Coverage-guided Model Generation For Deep Learning Library Testing](https://mlsanity.com/paper/2208.01508/) (2022)
- [Clarifying Semantics of In-Context Examples for Unit Test Generation](https://mlsanity.com/paper/2510.01994/) (2025)
- [Why Does the LLM Stop Computing: An Empirical Study of User-Reported Failures in Open-Source LLMs](https://mlsanity.com/paper/2601.13655/) (2026)
- [Where Agent Frameworks Fall Short: Examining Functional Challenges and Usability Concerns](https://mlsanity.com/paper/2602.21806/) (2026)
- [Unraveling Code Clone Dynamics in Deep Learning Frameworks](https://mlsanity.com/paper/2404.17046/) (2024)
- [An Empirical Study on Bugs Inside PyTorch: A Replication Study](https://mlsanity.com/paper/2307.13777/) (2023)
- [Studying the Impact of TensorFlow and PyTorch Bindings on Machine Learning Software Quality](https://mlsanity.com/paper/2407.05466/) (2024)
- [LibVulnWatch: A Deep Assessment Agent System and Leaderboard for Uncovering Hidden Vulnerabilities in Open-Source AI Libraries](https://mlsanity.com/paper/2505.08842/) (2025)
- [An Empirical Study of Bugs in Data Visualization Libraries](https://mlsanity.com/paper/2506.15084/) (2025)
- [Why Attention Fails: A Taxonomy of Faults in Attention-Based Neural Networks](https://mlsanity.com/paper/2508.04925/) (2025)
- [Comparative analysis of real bugs in open-source Machine Learning projects -- A Registered Report](https://mlsanity.com/paper/2209.09932/) (2022)
- [Understanding Bugs in Multi-Language Deep Learning Frameworks](https://mlsanity.com/paper/2303.02695/) (2023)
- [A Comprehensive Empirical Study of Bugs in Open-Source Federated Learning Frameworks](https://mlsanity.com/paper/2308.05014/) (2023)
- [On the Mistaken Assumption of Interchangeable Deep Reinforcement Learning Implementations](https://mlsanity.com/paper/2503.22575/) (2025)
- [AutoEmpirical: LLM-Based Automated Research for Empirical Software Fault Analysis](https://mlsanity.com/paper/2510.04997/) (2025)
- [DepRadar: Agentic Coordination for Context Aware Defect Impact Analysis in Deep Learning Libraries](https://mlsanity.com/paper/2601.09440/) (2026)
- [Ekka: Automated Diagnosis of Silent Errors in LLM Inference](https://mlsanity.com/paper/2606.04594/) (2026)
- [From AI Technical Debt to Agentic Technical Debt: A Systematic Mapping of Root Causes and Manifestations in Agentic AI Systems](https://mlsanity.com/paper/2608.01001/) (2026)
- [Silent Failures Beyond the 32-Bit Index Range: A Differential Characterization of Large-Tensor Matrix Multiplication in PyTorch's MPS Backend](https://mlsanity.com/paper/2609.22991/) (2026)

## Abstract (arXiv)

DL frameworks are the basis of constructing all DL programs and models, and thus their bugs could lead to the unexpected behaviors of any DL program or model relying on them. Such a wide effect demonstrates the necessity and importance of guaranteeing DL frameworks' quality. Understanding the characteristics of DL framework bugs is a fundamental step for this quality assurance task, facilitating designing effective bug detection and debugging approaches. Hence, in this work we conduct the most large-scale study on 1,000 bugs from four popular and diverse DL frameworks (i.e., TensorFlow, PyTorch, MXNet, and DL4J). By analyzing the root causes and symptoms of DL framework bugs associated with 5 components decomposed from DL frameworks, as well as measuring test coverage achieved by three state-of-the-art testing techniques, we obtain 12 major findings for the comprehensive understanding of DL framework bugs and the current status of existing DL framework testing practice, and then provide a series of actionable guidelines for better DL framework bug detection and debugging. Finally, based on the guidelines, we design and implement a prototype DL-framework testing tool, called TenFuzz, which is evaluated to be effective and finds 3 unknown bugs on the latest TensorFlow framework in a preliminary study, indicating the significance of our guidelines.