# Let's Play Again: Variability of Deep Reinforcement Learning Agents in Atari Environments

arXiv: https://arxiv.org/abs/1904.06312 · HTML: https://mlsanity.com/paper/1904.06312/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Apr 2019.

## Built on by (most cited first, 11 of 11)

- [Deep Reinforcement Learning at the Edge of the Statistical Precipice](https://mlsanity.com/paper/2108.13264/) (2021)
- [Is High Variance Unavoidable in RL? A Case Study in Continuous Control](https://mlsanity.com/paper/2110.11222/) (2021)
- [RAPTOR: A Foundation Policy for Quadrotor Control](https://mlsanity.com/paper/2509.11481/) (2025)
- [Replicable Reinforcement Learning with Linear Function Approximation](https://mlsanity.com/paper/2509.08660/) (2025)
- [Deep Reinforcement Learning for Cryptocurrency Trading: Practical Approach to Address Backtest Overfitting](https://mlsanity.com/paper/2209.05559/) (2022)
- [Performance Variation in Deep Reinforcement Learning](https://mlsanity.com/paper/2606.06746/) (2026)
- [Behavior-Consistent Deep Reinforcement Learning](https://mlsanity.com/paper/2605.21214/) (2026)
- [Measuring Interventional Robustness in Reinforcement Learning](https://mlsanity.com/paper/2209.09058/) (2022)
- [Plume: A Framework for High Performance Deep RL Network Controllers via Prioritized Trace Sampling](https://mlsanity.com/paper/2302.12403/) (2023)
- [From Optimal Policies to Individual Differences: Rethinking Reinforcement Learning for Biology](https://mlsanity.com/paper/2607.16542/) (2026)
- [Separating Decision Time from Decision Quality in the Real-Time Gap of Distilled Deciders: Evidence from a Game and a Conveyor Simulator](https://mlsanity.com/paper/2610.04810/) (2026)

## Abstract (arXiv)

Reproducibility in reinforcement learning is challenging: uncontrolled stochasticity from many sources, such as the learning algorithm, the learned policy, and the environment itself have led researchers to report the performance of learned agents using aggregate metrics of performance over multiple random seeds for a single environment. Unfortunately, there are still pernicious sources of variability in reinforcement learning agents that make reporting common summary statistics an unsound metric for performance. Our experiments demonstrate the variability of common agents used in the popular OpenAI Baselines repository. We make the case for reporting post-training agent performance as a distribution, rather than a point estimate.