# Is Deep Reinforcement Learning Really Superhuman on Atari? Leveling the playing field

arXiv: https://arxiv.org/abs/1908.04683 · HTML: https://mlsanity.com/paper/1908.04683/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Aug 2019.

## Built on by (most cited first, 15 of 15)

- [End-to-End Urban Driving by Imitating a Reinforcement Learning Coach](https://mlsanity.com/paper/2108.08265/) (2021)
- [GRI: General Reinforced Imitation and its Application to Vision-Based Autonomous Driving](https://mlsanity.com/paper/2111.08575/) (2021)
- [OCAtari: Object-Centric Atari 2600 Reinforcement Learning Environments](https://mlsanity.com/paper/2306.08649/) (2023)
- [Eventual Discounting Temporal Logic Counterfactual Experience Replay](https://mlsanity.com/paper/2303.02135/) (2023)
- [Open RL Benchmark: Comprehensive Tracked Experiments for Reinforcement Learning](https://mlsanity.com/paper/2402.03046/) (2024)
- [Shielding Atari Games with Bounded Prescience](https://mlsanity.com/paper/2101.08153/) (2021)
- [A Review for Deep Reinforcement Learning in Atari:Benchmarks, Challenges, and Solutions](https://mlsanity.com/paper/2112.04145/) (2021)
- [Generalized Data Distribution Iteration](https://mlsanity.com/paper/2206.03192/) (2022)
- [HackAtari: Atari Learning Environments for Robust and Continual Reinforcement Learning](https://mlsanity.com/paper/2406.03997/) (2024)
- [GDI: Rethinking What Makes Reinforcement Learning Different From Supervised Learning](https://mlsanity.com/paper/2106.06232/) (2021)
- [Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PC](https://mlsanity.com/paper/2411.03820/) (2024)
- [CASA: Bridging the Gap between Policy Improvement and Policy Evaluation with Conflict Averse Policy Iteration](https://mlsanity.com/paper/2105.03923/) (2021)
- [More Efficient Exploration with Symbolic Priors on Action Sequence Equivalences](https://mlsanity.com/paper/2110.10632/) (2021)
- [A General, Evolution-Inspired Reward Function for Social Robotics](https://mlsanity.com/paper/2202.00617/) (2022)
- [SpeedrunBench: Challenging LLM Agents with Video Game Speedrunning](https://mlsanity.com/paper/2610.08076/) (2026)

## Abstract (arXiv)

Consistent and reproducible evaluation of Deep Reinforcement Learning (DRL) is not straightforward. In the Arcade Learning Environment (ALE), small changes in environment parameters such as stochasticity or the maximum allowed play time can lead to very different performance. In this work, we discuss the difficulties of comparing different agents trained on ALE. In order to take a step further towards reproducible and comparable DRL, we introduce SABER, a Standardized Atari BEnchmark for general Reinforcement learning algorithms. Our methodology extends previous recommendations and contains a complete set of environment parameters as well as train and test procedures. We then use SABER to evaluate the current state of the art, Rainbow. Furthermore, we introduce a human world records baseline, and argue that previous claims of expert or superhuman performance of DRL might not be accurate. Finally, we propose Rainbow-IQN by extending Rainbow with Implicit Quantile Networks (IQN) leading to new state-of-the-art performance. Source code is available for reproducibility.