# Improving the Reproducibility of Deep Learning Software: An Initial Investigation through a Case Study Analysis

arXiv: https://arxiv.org/abs/2505.03165 · HTML: https://mlsanity.com/paper/2505.03165/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 56 references that are in the map.
- First posted on arXiv 6 May 2025.

## Builds on (most cited first, 22 of 22)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [RandAugment: Practical automated data augmentation with a reduced search space](https://mlsanity.com/paper/1909.13719/) (2019)
- [Improved Regularization of Convolutional Neural Networks with Cutout](https://mlsanity.com/paper/1708.04552/) (2017)
- [How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers](https://mlsanity.com/paper/2106.10270/) (2021)
- [DeiT III: Revenge of the ViT](https://mlsanity.com/paper/2204.07118/) (2022)
- [Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)](https://mlsanity.com/paper/2003.12206/) (2020)
- [EMNIST: an extension of MNIST to handwritten letters](https://mlsanity.com/paper/1702.05373/) (2017)
- [YOLO-Pose: Enhancing YOLO for Multi Person Pose Estimation Using Object Keypoint Similarity Loss](https://mlsanity.com/paper/2204.06806/) (2022)
- [B-CNN: Branch Convolutional Neural Network for Hierarchical Classification](https://mlsanity.com/paper/1709.09890/) (2017)
- [Tree-CNN: A Hierarchical Deep Convolutional Neural Network for Incremental Learning](https://mlsanity.com/paper/1802.05800/) (2018)
- [An introduction to Docker for reproducible research, with examples from the R environment](https://mlsanity.com/paper/1410.0846/) (2014)
- [torchdistill: A Modular, Configuration-Driven Framework for Knowledge Distillation](https://mlsanity.com/paper/2011.12913/) (2020)
- [Reproducibility in Machine Learning-Driven Research](https://mlsanity.com/paper/2307.10320/) (2023)
- [Improving the repeatability of deep learning models with Monte Carlo dropout](https://mlsanity.com/paper/2202.07562/) (2022)
- [Challenges and Practices of Deep Learning Model Reengineering: A Case Study on Computer Vision](https://mlsanity.com/paper/2303.07476/) (2023)
- [Reusing Deep Learning Models: Challenges and Directions in Software Engineering](https://mlsanity.com/paper/2404.16688/) (2024)
- [A Partial Replication of MaskFormer in TensorFlow on TPUs for the TensorFlow Model Garden](https://mlsanity.com/paper/2404.18801/) (2024)

## Built on by (most cited first, 3 of 3)

- [Introducing Background Temperature to Characterise Hidden Randomness in Large Language Models](https://mlsanity.com/paper/2604.22411/) (2026)
- [A Comprehensive Evaluation of YOLO-based Deer Detection Performance on Edge Devices](https://mlsanity.com/paper/2509.20318/) (2025)
- [Position: Let's Strengthen Verifiability If We Can't Enforce Reproducibility](https://mlsanity.com/paper/2609.35854/) (2026)

## Abstract (arXiv)

The field of deep learning has witnessed significant breakthroughs, spanning various applications, and fundamentally transforming current software capabilities. However, alongside these advancements, there have been increasing concerns about reproducing the results of these deep learning methods. This is significant because reproducibility is the foundation of reliability and validity in software development, particularly in the rapidly evolving domain of deep learning. The difficulty of reproducibility may arise due to several reasons, including having differences from the original execution environment, incompatible software libraries, proprietary data and source code, lack of transparency, and the stochastic nature in some software. A study conducted by the Nature journal reveals that more than 70% of researchers failed to reproduce other researchers experiments and over 50% failed to reproduce their own experiments. Irreproducibility of deep learning poses significant challenges for researchers and practitioners. To address these concerns, this paper presents a systematic approach at analyzing and improving the reproducibility of deep learning models by demonstrating these guidelines using a case study. We illustrate the patterns and anti-patterns involved with these guidelines for improving the reproducibility of deep learning models. These guidelines encompass establishing a methodology to replicate the original software environment, implementing end-to-end training and testing algorithms, disclosing architectural designs, and enhancing transparency in data processing and training pipelines. We also conduct a sensitivity analysis to understand the model performance across diverse conditions. By implementing these strategies, we aim to bridge the gap between research and practice, so that innovations in deep learning can be effectively reproduced and deployed within software.