# Towards Enhancing the Reproducibility of Deep Learning Bugs: An Empirical Study

arXiv: https://arxiv.org/abs/2401.03069 · HTML: https://mlsanity.com/paper/2401.03069/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 80 references that are in the map.
- First posted on arXiv 5 Jan 2024.

## Builds on (most cited first, 15 of 15)

- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [DeepXplore: Automated Whitebox Testing of Deep Learning Systems](https://mlsanity.com/paper/1705.06640/) (2017)
- [DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems](https://mlsanity.com/paper/1803.07519/) (2018)
- [Taxonomy of Real Faults in Deep Learning Systems](https://mlsanity.com/paper/1910.11015/) (2019)
- [A Comprehensive Study on Deep Learning Bug Characteristics](https://mlsanity.com/paper/1906.01388/) (2019)
- [On Testing Machine Learning Programs](https://mlsanity.com/paper/1812.02257/) (2018)
- [Silent Bugs in Deep Learning Frameworks: An Empirical Study of Keras and TensorFlow](https://mlsanity.com/paper/2112.13314/) (2021)
- [Towards Training Reproducible Deep Learning Models](https://mlsanity.com/paper/2202.02326/) (2022)
- [Benchmarking State-of-the-Art Deep Learning Software Tools](https://mlsanity.com/paper/1608.07249/) (2016)
- [On the Replicability and Reproducibility of Deep Learning in Software Engineering](https://mlsanity.com/paper/2006.14244/) (2020)
- [Bugs in Machine Learning-based Systems: A Faultload Benchmark](https://mlsanity.com/paper/2206.12311/) (2022)
- [On Reporting Performance and Accuracy Bugs for Deep Learning Frameworks: An Exploratory Study from GitHub](https://mlsanity.com/paper/2204.07893/) (2022)
- [Why are Some Bugs Non-Reproducible? An Empirical Investigation using Data Fusion](https://mlsanity.com/paper/2108.05316/) (2021)
- [The Reproducibility of Programming-Related Issues in Stack Overflow Questions](https://mlsanity.com/paper/2111.12204/) (2021)

## Built on by (most cited first, 6 of 6)

- [Characterizing Faults in Agentic AI: A Taxonomy of Types, Symptoms, and Root Causes](https://mlsanity.com/paper/2603.06847/) (2026)
- [Past, Present, and Future of Bug Tracking in the Generative AI Era](https://mlsanity.com/paper/2510.08005/) (2025)
- [Feature-Guided Analysis of Neural Networks: A Replication Study](https://mlsanity.com/paper/2511.00052/) (2025)
- [An Empirical Study of OpenAI API Discussions on Stack Overflow](https://mlsanity.com/paper/2505.04084/) (2025)
- [Imitation Game: Reproducing Deep Learning Bugs Leveraging an Intelligent Agent](https://mlsanity.com/paper/2512.14990/) (2025)
- [Real Faults in Deep Learning Fault Benchmarks: How Real Are They?](https://mlsanity.com/paper/2412.16336/) (2024)

## Abstract (arXiv)

Context: Deep learning has achieved remarkable progress in various domains. However, like any software system, deep learning systems contain bugs, some of which can have severe impacts, as evidenced by crashes involving autonomous vehicles. Despite substantial advancements in deep learning techniques, little research has focused on reproducing deep learning bugs, which is an essential step for their resolution. Existing literature suggests that only 3% of deep learning bugs are reproducible, underscoring the need for further research. Objective: This paper examines the reproducibility of deep learning bugs. We identify edit actions and useful information that could improve the reproducibility of deep learning bugs. Method: First, we construct a dataset of 668 deep-learning bugs from Stack Overflow and GitHub across three frameworks and 22 architectures. Second, out of the 668 bugs, we select 165 bugs using stratified sampling and attempt to determine their reproducibility. While reproducing these bugs, we identify edit actions and useful information for their reproduction. Third, we used the Apriori algorithm to identify useful information and edit actions required to reproduce specific types of bugs. Finally, we conducted a user study involving 22 developers to assess the effectiveness of our findings in real-life settings. Results: We successfully reproduced 148 out of 165 bugs attempted. We identified ten edit actions and five useful types of component information that can help us reproduce the deep learning bugs. With the help of our findings, the developers were able to reproduce 22.92% more bugs and reduce their reproduction time by 24.35%. Conclusions: Our research addresses the critical issue of deep learning bug reproducibility. Practitioners and researchers can leverage our findings to improve deep learning bug reproducibility.