# Comparative analysis of real bugs in open-source Machine Learning projects -- A Registered Report

arXiv: https://arxiv.org/abs/2209.09932 · HTML: https://mlsanity.com/paper/2209.09932/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 32 references that are in the map.
- First posted on arXiv 20 Sep 2022.

## Builds on (most cited first, 10 of 10)

- [Taxonomy of Real Faults in Deep Learning Systems](https://mlsanity.com/paper/1910.11015/) (2019)
- [A Comprehensive Study on Deep Learning Bug Characteristics](https://mlsanity.com/paper/1906.01388/) (2019)
- [A Systematic Study of Online Class Imbalance Learning with Concept Drift](https://mlsanity.com/paper/1703.06683/) (2017)
- [Toward Understanding Deep Learning Framework Bugs](https://mlsanity.com/paper/2203.04026/) (2022)
- [Silent Bugs in Deep Learning Frameworks: An Empirical Study of Keras and TensorFlow](https://mlsanity.com/paper/2112.13314/) (2021)
- [Verification for Machine Learning, Autonomy, and Neural Networks Survey](https://mlsanity.com/paper/1810.01989/) (2018)
- [DeepDiagnosis: Automatically Diagnosing Faults and Recommending Actionable Fixes in Deep Learning Programs](https://mlsanity.com/paper/2112.04036/) (2021)
- [Analysis and Detection of Information Types of Open Source Software Issue Discussions](https://mlsanity.com/paper/1902.07093/) (2019)
- [A large-scale comparative analysis of Coding Standard conformance in Open-Source Data Science projects](https://mlsanity.com/paper/2007.08978/) (2020)
- [Memory and Resource Leak Defects and their Repairs in Java Projects](https://mlsanity.com/paper/1810.00101/) (2018)

## Abstract (arXiv)

Background: Machine Learning (ML) systems rely on data to make predictions, the systems have many added components compared to traditional software systems such as the data processing pipeline, serving pipeline, and model training. Existing research on software maintenance has studied the issue-reporting needs and resolution process for different types of issues, such as performance and security issues. However, ML systems have specific classes of faults, and reporting ML issues requires domain-specific information. Because of the different characteristics between ML and traditional Software Engineering systems, we do not know to what extent the reporting needs are different, and to what extent these differences impact the issue resolution process. Objective: Our objective is to investigate whether there is a discrepancy in the distribution of resolution time between ML and non-ML issues and whether certain categories of ML issues require a longer time to resolve based on real issue reports in open-source applied ML projects. We further investigate the size of fix of ML issues and non-ML issues. Method: We extract issues reports, pull requests and code files in recent active applied ML projects from Github, and use an automatic approach to filter ML and non-ML issues. We manually label the issues using a known taxonomy of deep learning bugs. We measure the resolution time and size of fix of ML and non-ML issues on a controlled sample and compare the distributions for each category of issue.