# Towards Training Reproducible Deep Learning Models

arXiv: https://arxiv.org/abs/2202.02326 · HTML: https://mlsanity.com/paper/2202.02326/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 66 references that are in the map.
- First posted on arXiv 4 Feb 2022.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Datasheets for Datasets](https://mlsanity.com/paper/1803.09010/) (2018)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Deep Reinforcement Learning that Matters](https://mlsanity.com/paper/1709.06560/) (2017)
- [Deep Speech 2: End-to-End Speech Recognition in English and Mandarin](https://mlsanity.com/paper/1512.02595/) (2015)
- [A Survey of Deep Learning Techniques for Autonomous Driving](https://mlsanity.com/paper/1910.07738/) (2019)
- [Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)](https://mlsanity.com/paper/2003.12206/) (2020)
- [cuDNN: Efficient Primitives for Deep Learning](https://mlsanity.com/paper/1410.0759/) (2014)
- [Toward Trustworthy AI Development: Mechanisms for Supporting Verifiable Claims](https://mlsanity.com/paper/2004.07213/) (2020)
- [DeepGauge: Multi-Granularity Testing Criteria for Deep Learning Systems](https://mlsanity.com/paper/1803.07519/) (2018)
- [A Step Toward Quantifying Independently Reproducible Machine Learning Research](https://mlsanity.com/paper/1909.06674/) (2019)
- [How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments](https://mlsanity.com/paper/1806.08295/) (2018)
- [An Empirical Study towards Characterizing Deep Learning Development and Deployment across Different Frameworks and Platforms](https://mlsanity.com/paper/1909.06727/) (2019)
- [Importance-Driven Deep Learning System Testing](https://mlsanity.com/paper/2002.03433/) (2020)
- [On the Replicability and Reproducibility of Deep Learning in Software Engineering](https://mlsanity.com/paper/2006.14244/) (2020)
- [Asset Management in Machine Learning: A Survey](https://mlsanity.com/paper/2102.06919/) (2021)
- [Building a Reproducible Machine Learning Pipeline](https://mlsanity.com/paper/1810.04570/) (2018)
- [Detecting Deep Neural Network Defects with Data Flow Analysis](https://mlsanity.com/paper/1909.02190/) (2019)

## Built on by (most cited first, 23 of 23)

- [Reproducibility in Machine Learning-based Research: Overview, Barriers and Drivers](https://mlsanity.com/paper/2406.14325/) (2024)
- [A Survey on Stability of Learning with Limited Labelled Data and its Sensitivity to the Effects of Randomness](https://mlsanity.com/paper/2312.01082/) (2023)
- [Challenges and Practices of Deep Learning Model Reengineering: A Case Study on Computer Vision](https://mlsanity.com/paper/2303.07476/) (2023)
- [Pitfalls in Language Models for Code Intelligence: A Taxonomy and Survey](https://mlsanity.com/paper/2310.17903/) (2023)
- [Reproducibility of Machine Learning: Terminology, Recommendations and Open Issues](https://mlsanity.com/paper/2302.12691/) (2023)
- [Watson: A Cognitive Observability Framework for the Reasoning of LLM-Powered Agents](https://mlsanity.com/paper/2411.03455/) (2024)
- [Your Compiler is Backdooring Your Model: Understanding and Exploiting Compilation Inconsistency Vulnerabilities in Deep Learning Compilers](https://mlsanity.com/paper/2509.11173/) (2025)
- [Towards Enhancing the Reproducibility of Deep Learning Bugs: An Empirical Study](https://mlsanity.com/paper/2401.03069/) (2024)
- [Confronting the Reproducibility Crisis: A Case Study of Challenges in Cybersecurity AI](https://mlsanity.com/paper/2405.18753/) (2024)
- [Investigating the Impact of Randomness on Reproducibility in Computer Vision: A Study on Applications in Civil Engineering and Medicine](https://mlsanity.com/paper/2410.02806/) (2024)
- [Attention is Not Always What You Need: Towards Efficient Classification of Domain-Specific Text](https://mlsanity.com/paper/2303.17786/) (2023)
- [Investigating Reproducibility in Deep Learning-Based Software Fault Prediction](https://mlsanity.com/paper/2402.05645/) (2024)
- [Beyond the Black Box: Do More Complex Deep Learning Models Provide Superior XAI Explanations?](https://mlsanity.com/paper/2405.08658/) (2024)
- [Studying the Impact of TensorFlow and PyTorch Bindings on Machine Learning Software Quality](https://mlsanity.com/paper/2407.05466/) (2024)
- [Bit-Accurate Modeling of GPU Matrix Multiply-Accumulate Units: Demystifying Numerical Discrepancy and Accuracy](https://mlsanity.com/paper/2511.10909/) (2025)
- [Variance of ML-based software fault predictors: are we really improving fault prediction?](https://mlsanity.com/paper/2310.17264/) (2023)
- [Reproducibility and Geometric Intrinsic Dimensionality: An Investigation on Graph Neural Network Research](https://mlsanity.com/paper/2403.08438/) (2024)
- [Software Performance Engineering for Foundation Model-Powered Software](https://mlsanity.com/paper/2411.09580/) (2024)
- [An Empirical Framework for Evaluating Semantic Preservation Using Hugging Face](https://mlsanity.com/paper/2512.07983/) (2025)
- [A Model-Driven Lossless Compression Algorithm Resistant to Mismatch](https://mlsanity.com/paper/2601.17684/) (2026)
- [Bit-Identical Medical Deep Learning via Structured Orthogonal Initialization](https://mlsanity.com/paper/2603.28040/) (2026)
- [Reproducibility is the New Copyleft: Defining AGI-oriented Reproducible Builds](https://mlsanity.com/paper/2606.03019/) (2026)
- [OVIG: Optimistic Verification of AI Training Integrity via Gradient Signals](https://mlsanity.com/paper/2606.21045/) (2026)

## Abstract (arXiv)

Reproducibility is an increasing concern in Artificial Intelligence (AI), particularly in the area of Deep Learning (DL). Being able to reproduce DL models is crucial for AI-based systems, as it is closely tied to various tasks like training, testing, debugging, and auditing. However, DL models are challenging to be reproduced due to issues like randomness in the software (e.g., DL algorithms) and non-determinism in the hardware (e.g., GPU). There are various practices to mitigate some of the aforementioned issues. However, many of them are either too intrusive or can only work for a specific usage context. In this paper, we propose a systematic approach to training reproducible DL models. Our approach includes three main parts: (1) a set of general criteria to thoroughly evaluate the reproducibility of DL models for two different domains, (2) a unified framework which leverages a record-and-replay technique to mitigate software-related randomness and a profile-and-patch technique to control hardware-related non-determinism, and (3) a reproducibility guideline which explains the rationales and the mitigation strategies on conducting a reproducible training process for DL models. Case study results show our approach can successfully reproduce six open source and one commercial DL models.