# On Reporting Performance and Accuracy Bugs for Deep Learning Frameworks: An Exploratory Study from GitHub

arXiv: https://arxiv.org/abs/2204.07893 · HTML: https://mlsanity.com/paper/2204.07893/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Apr 2022.

## Built on by (most cited first, 7 of 7)

- [Bug Characterization in Machine Learning-based Systems](https://mlsanity.com/paper/2307.14512/) (2023)
- [Towards Enhancing the Reproducibility of Deep Learning Bugs: An Empirical Study](https://mlsanity.com/paper/2401.03069/) (2024)
- [Learning Software Bug Reports: A Systematic Literature Review](https://mlsanity.com/paper/2507.04422/) (2025)
- [Unraveling Code Clone Dynamics in Deep Learning Frameworks](https://mlsanity.com/paper/2404.17046/) (2024)
- [Multifaceted Hierarchical Report Identification for Non-Functional Bugs in Deep Learning Frameworks](https://mlsanity.com/paper/2210.01855/) (2022)
- [Human-Machine Co-Boosted Bug Report Identification with Mutualistic Neural Active Learning](https://mlsanity.com/paper/2604.18862/) (2026)
- [Configuring RAG Systems: What Breaks, Where Originates, and How to Fix](https://mlsanity.com/paper/2610.18567/) (2026)

## Abstract (arXiv)

The tremendous success of Deep Learning (DL) has significantly boosted the number of open-sourced DL frameworks hosted on GitHub. Among others, performance and accuracy bugs are critical factors that affect the reputation of these DL frameworks, therefore understanding the practice of discovering and investigating them for DL is important. In this paper, we conduct an exploratory study on the nature of reporting performance and accuracy bugs bugs for DL frameworks, aiming to improve our knowledge on this topic. Our study covers 10 most popular open-sourced DL frameworks on GitHub (e.g., TensorFlow, Keras, and PyTorch), based on which we sample 664 representative performance and accuracy bugs bug reports out of a total population of 22,522. Through systematic analysis of these samples, our key findings are: (1) low speed is the primary reason that a performance bug related report is submitted but we see no consistent pattern for accuracy related ones; (2) most of the reports are about issues encountered in the training stage; (3) only a small proportion of the reports provide insufficient information to investigate; (4) the majority of the performance and accuracy bugs bug reports (from 69% to 100%) are not related to the actual bug or regarded as unclassified; (5) around 50% of the performance and accuracy bug reports, which indeed reveal bugs, are not resolved by direct patches. Deriving from the above, we discuss a set of actionable implications to the researchers, maintainers, and report submitters on this subject. To promote open science, the labeled dataset has been made publicly available at https://tinyurl.com/4x3tap9w.