# Deterministic Implementations for Reproducibility in Deep Reinforcement Learning

arXiv: https://arxiv.org/abs/1809.05676 · HTML: https://mlsanity.com/paper/1809.05676/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Sep 2018.

## Built on by (most cited first, 19 of 19)

- [Deep Reinforcement Learning at the Edge of the Statistical Precipice](https://mlsanity.com/paper/2108.13264/) (2021)
- [Reproducibility in Machine Learning-based Research: Overview, Barriers and Drivers](https://mlsanity.com/paper/2406.14325/) (2024)
- [Replicable Reinforcement Learning](https://mlsanity.com/paper/2305.15284/) (2023)
- [The worst of both worlds: A comparative analysis of errors in learning from data in psychology and machine learning](https://mlsanity.com/paper/2203.06498/) (2022)
- [Reproducibility in Machine Learning-Driven Research](https://mlsanity.com/paper/2307.10320/) (2023)
- [Generalization Guarantees for Neural Architecture Search with Train-Validation Split](https://mlsanity.com/paper/2104.14132/) (2021)
- [Synthesizing Irreproducibility in Deep Networks](https://mlsanity.com/paper/2102.10696/) (2021)
- [mlf-core: a framework for deterministic machine learning](https://mlsanity.com/paper/2104.07651/) (2021)
- [The NCI Imaging Data Commons as a platform for reproducible research in computational pathology](https://mlsanity.com/paper/2303.09354/) (2023)
- [Double Actor-Critic with TD Error-Driven Regularization in Reinforcement Learning](https://mlsanity.com/paper/2409.19231/) (2024)
- [Real World Large Scale Recommendation Systems Reproducibility and Smooth Activations](https://mlsanity.com/paper/2202.06499/) (2022)
- [UR2: Unify RAG and Reasoning through Reinforcement Learning](https://mlsanity.com/paper/2508.06165/) (2025)
- [Studying the Impact of TensorFlow and PyTorch Bindings on Machine Learning Software Quality](https://mlsanity.com/paper/2407.05466/) (2024)
- [A Comparison of the Delta Method and the Bootstrap in Deep Learning Classification](https://mlsanity.com/paper/2107.01606/) (2021)
- [Variance of ML-based software fault predictors: are we really improving fault prediction?](https://mlsanity.com/paper/2310.17264/) (2023)
- [MuFlex: A Scalable, Physics-based Platform for Multi-Building Flexibility Analysis and Coordination](https://mlsanity.com/paper/2508.13532/) (2025)
- [Hyperparameter Optimization and Reproducibility in Deep Learning Model Training](https://mlsanity.com/paper/2510.15164/) (2025)
- [Numerical Instability and Chaos: Quantifying the Unpredictability of Large Language Models](https://mlsanity.com/paper/2604.13206/) (2026)
- [Omega-S: A Functional Resilience Index for LLM Fine-Tuning](https://mlsanity.com/paper/2608.03887/) (2026)

## Abstract (arXiv)

While deep reinforcement learning (DRL) has led to numerous successes in recent years, reproducing these successes can be extremely challenging. One reproducibility challenge particularly relevant to DRL is nondeterminism in the training process, which can substantially affect the results. Motivated by this challenge, we study the positive impacts of deterministic implementations in eliminating nondeterminism in training. To do so, we consider the particular case of the deep Q-learning algorithm, for which we produce a deterministic implementation by identifying and controlling all sources of nondeterminism in the training process. One by one, we then allow individual sources of nondeterminism to affect our otherwise deterministic implementation, and measure the impact of each source on the variance in performance. We find that individual sources of nondeterminism can substantially impact the performance of agent, illustrating the benefits of deterministic implementations. In addition, we also discuss the important role of deterministic implementations in achieving exact replicability of results.