# Investigating Reproducibility in Deep Learning-Based Software Fault Prediction

arXiv: https://arxiv.org/abs/2402.05645 · HTML: https://mlsanity.com/paper/2402.05645/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 68 references that are in the map.
- First posted on arXiv 8 Feb 2024.

## Builds on (most cited first, 13 of 13)

- [Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)](https://mlsanity.com/paper/2003.12206/) (2020)
- [Recent Trends in Deep Learning Based Natural Language Processing](https://mlsanity.com/paper/1708.02709/) (2017)
- [Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approaches](https://mlsanity.com/paper/1907.06902/) (2019)
- [A Step Toward Quantifying Independently Reproducible Machine Learning Research](https://mlsanity.com/paper/1909.06674/) (2019)
- [Vulnerability Detection with Fine-grained Interpretations](https://mlsanity.com/paper/2106.10478/) (2021)
- [A Survey on Deep Learning for Software Engineering](https://mlsanity.com/paper/2011.14597/) (2020)
- [A Systematic Literature Review on the Use of Deep Learning in Software Engineering Research](https://mlsanity.com/paper/2009.06520/) (2020)
- [Sorting and Transforming Program Repair Ingredients via Deep Learning Code Similarities](https://mlsanity.com/paper/1707.04742/) (2017)
- [Towards Training Reproducible Deep Learning Models](https://mlsanity.com/paper/2202.02326/) (2022)
- [On the Replicability and Reproducibility of Deep Learning in Software Engineering](https://mlsanity.com/paper/2006.14244/) (2020)
- [VulCurator: A Vulnerability-Fixing Commit Detector](https://mlsanity.com/paper/2209.03260/) (2022)
- [The Effect of Third Party Implementations on Reproducibility](https://mlsanity.com/paper/2307.14956/) (2023)
- [Deep Learning in Software Engineering](https://mlsanity.com/paper/1805.04825/) (2018)

## Built on by (most cited first, 1 of 1)

- [Reproducibility of Machine Learning-Based Fault Detection and Diagnosis for HVAC Systems in Buildings: An Empirical Study](https://mlsanity.com/paper/2508.00880/) (2025)

## Abstract (arXiv)

Over the past few years, deep learning methods have been applied for a wide range of Software Engineering (SE) tasks, including in particular for the important task of automatically predicting and localizing faults in software. With the rapid adoption of increasingly complex machine learning models, it however becomes more and more difficult for scholars to reproduce the results that are reported in the literature. This is in particular the case when the applied deep learning models and the evaluation methodology are not properly documented and when code and data are not shared. Given some recent -- and very worrying -- findings regarding reproducibility and progress in other areas of applied machine learning, the goal of this work is to analyze to what extent the field of software engineering, in particular in the area of software fault prediction, is plagued by similar problems. We have therefore conducted a systematic review of the current literature and examined the level of reproducibility of 56 research articles that were published between 2019 and 2022 in top-tier software engineering conferences. Our analysis revealed that scholars are apparently largely aware of the reproducibility problem, and about two thirds of the papers provide code for their proposed deep learning models. However, it turned out that in the vast majority of cases, crucial elements for reproducibility are missing, such as the code of the compared baselines, code for data pre-processing or code for hyperparameter tuning. In these cases, it therefore remains challenging to exactly reproduce the results in the current research literature. Overall, our meta-analysis therefore calls for improved research practices to ensure the reproducibility of machine-learning based research.