# Reliable validation of Reinforcement Learning Benchmarks

arXiv: https://arxiv.org/abs/2203.01075 · HTML: https://mlsanity.com/paper/2203.01075/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 56 references that are in the map.
- First posted on arXiv 2 Mar 2022.

## Builds on (most cited first, 24 of 24)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Deep Reinforcement Learning that Matters](https://mlsanity.com/paper/1709.06560/) (2017)
- [Green AI](https://mlsanity.com/paper/1907.10597/) (2019)
- [Carbon Emissions and Large Neural Network Training](https://mlsanity.com/paper/2104.10350/) (2021)
- [Leveraging Procedural Generation to Benchmark Reinforcement Learning](https://mlsanity.com/paper/1912.01588/) (2019)
- [Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)](https://mlsanity.com/paper/2003.12206/) (2020)
- [Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Control](https://mlsanity.com/paper/1708.04133/) (2017)
- [Increasing Generality in Machine Learning through Procedural Content Generation](https://mlsanity.com/paper/1911.13071/) (2019)
- [The MineRL 2020 Competition on Sample Efficient Reinforcement Learning using Human Priors](https://mlsanity.com/paper/2101.11071/) (2021)
- [An introduction to Docker for reproducible research, with examples from the R environment](https://mlsanity.com/paper/1410.0846/) (2014)
- [Deep Reinforcement Learning for General Video Game AI](https://mlsanity.com/paper/1806.02448/) (2018)
- [On the Replicability and Reproducibility of Deep Learning in Software Engineering](https://mlsanity.com/paper/2006.14244/) (2020)
- [Deep Reinforcement Learning](https://mlsanity.com/paper/1810.06339/) (2018)
- [Reproducibility in Evolutionary Computation](https://mlsanity.com/paper/2102.03380/) (2021)
- [Gym-Ignition: Reproducible Robotic Simulations for Reinforcement Learning](https://mlsanity.com/paper/1911.01715/) (2019)
- [Valuing Player Actions in Counter-Strike: Global Offensive](https://mlsanity.com/paper/2011.01324/) (2020)
- [Procedural Content Generation: Better Benchmarks for Transfer Reinforcement Learning](https://mlsanity.com/paper/2105.14780/) (2021)
- [rl_reach: Reproducible Reinforcement Learning Experiments for Robotic Reaching Tasks](https://mlsanity.com/paper/2102.04916/) (2021)
- [Evolutionarily-Curated Curriculum Learning for Deep Reinforcement Learning Agents](https://mlsanity.com/paper/1901.05431/) (2019)
- [The Dota 2 Bot Competition](https://mlsanity.com/paper/2103.02943/) (2021)
- [Snakes AI Competition 2020 and 2021 Report](https://mlsanity.com/paper/2108.05136/) (2021)

## Abstract (arXiv)

Reinforcement Learning (RL) is one of the most dynamic research areas in Game AI and AI as a whole, and a wide variety of games are used as its prominent test problems. However, it is subject to the replicability crisis that currently affects most algorithmic AI research. Benchmarking in Reinforcement Learning could be improved through verifiable results. There are numerous benchmark environments whose scores are used to compare different algorithms, such as Atari. Nevertheless, reviewers must trust that figures represent truthful values, as it is difficult to reproduce an exact training curve. We propose improving this situation by providing access to the original experimental data to validate study results. To that end, we rely on the concept of minimal traces. These allow re-simulation of action sequences in deterministic RL environments and, in turn, enable reviewers to verify, re-use, and manually inspect experimental results without needing large compute clusters. It also permits validation of presented reward graphs, an inspection of individual episodes, and re-use of result data (baselines) for proper comparison in follow-up papers. We offer plug-and-play code that works with Gym so that our measures fit well in the existing RL and reproducibility eco-system. Our approach is freely available, easy to use, and adds minimal overhead, as minimal traces allow a data compression ratio of up to ≈ 104:1 (94GB to 8MB for Atari Pong) compared to a regular MDP trace used in offline RL datasets. The paper presents proof-of-concept results for a variety of games.