# A Hitchhiker's Guide to Statistical Comparisons of Reinforcement Learning Algorithms

arXiv: https://arxiv.org/abs/1904.06979 · HTML: https://mlsanity.com/paper/1904.06979/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Apr 2019.

## Built on by (most cited first, 27 of 27)

- [Deep Reinforcement Learning at the Edge of the Statistical Precipice](https://mlsanity.com/paper/2108.13264/) (2021)
- [Towards a Standardised Performance Evaluation Protocol for Cooperative MARL](https://mlsanity.com/paper/2209.10485/) (2022)
- [Is High Variance Unavoidable in RL? A Case Study in Continuous Control](https://mlsanity.com/paper/2110.11222/) (2021)
- [Universal Morphology Control via Contextual Modulation](https://mlsanity.com/paper/2302.11070/) (2023)
- [Semantic HELM: A Human-Readable Memory for Reinforcement Learning](https://mlsanity.com/paper/2306.09312/) (2023)
- [A Domain-Agnostic Approach for Characterization of Lifelong Learning Systems](https://mlsanity.com/paper/2301.07799/) (2023)
- [Curriculum Learning for Safe Mapless Navigation](https://mlsanity.com/paper/2112.12490/) (2021)
- [Constrained Reinforcement Learning for Robotics via Scenario-Based Programming](https://mlsanity.com/paper/2206.09603/) (2022)
- [Learn the Time to Learn: Replay Scheduling in Continual Learning](https://mlsanity.com/paper/2209.08660/) (2022)
- [Policy Optimization in a Noisy Neighborhood: On Return Landscapes in Continuous Control](https://mlsanity.com/paper/2309.14597/) (2023)
- [Assessing SATNet's Ability to Solve the Symbol Grounding Problem](https://mlsanity.com/paper/2312.11522/) (2023)
- [Grounding Spatio-Temporal Language with Transformers](https://mlsanity.com/paper/2106.08858/) (2021)
- [Improving Deep Policy Gradients with Value Function Search](https://mlsanity.com/paper/2302.10145/) (2023)
- [Reinforcement Learning in Practice: Opportunities and Challenges](https://mlsanity.com/paper/2202.11296/) (2022)
- [Inferring Behavior-Specific Context Improves Zero-Shot Generalization in Reinforcement Learning](https://mlsanity.com/paper/2404.09521/) (2024)
- [Improving Policy Optimization via varepsilon-Retrain](https://mlsanity.com/paper/2406.08315/) (2024)
- [Improved Cooperation by Exploiting a Common Signal](https://mlsanity.com/paper/2102.02304/) (2021)
- [AI-driven Prices for Externalities and Sustainability in Production Markets](https://mlsanity.com/paper/2106.06060/) (2021)
- [AdaStop: adaptive statistical testing for sound comparisons of Deep RL agents](https://mlsanity.com/paper/2306.10882/) (2023)
- [Divide & Conquer Imitation Learning](https://mlsanity.com/paper/2204.07404/) (2022)
- [Learning An Active Inference Model of Driver Perception and Control: Application to Vehicle Car-Following](https://mlsanity.com/paper/2303.15201/) (2023)
- [Reinforcement Teaching](https://mlsanity.com/paper/2204.11897/) (2022)
- [Efficient Epistemic Uncertainty Estimation in Regression Ensemble Models Using Pairwise-Distance Estimators](https://mlsanity.com/paper/2308.13498/) (2023)
- [AtomComposer: Discovering Chemical Space from First Principles with Reinforcement Learning](https://mlsanity.com/paper/2605.28287/) (2026)
- [Blockwise Sequential Model Learning for Partially Observable Reinforcement Learning](https://mlsanity.com/paper/2112.05343/) (2021)
- [Boosting Robustness in Preference-Based Reinforcement Learning with Dynamic Sparsity](https://mlsanity.com/paper/2406.06495/) (2024)
- [Multi-action Tangled Program Graphs for Multi-task Reinforcement Learning with Continuous Control](https://mlsanity.com/paper/2604.25369/) (2026)

## Abstract (arXiv)

Consistently checking the statistical significance of experimental results is the first mandatory step towards reproducible science. This paper presents a hitchhiker's guide to rigorous comparisons of reinforcement learning algorithms. After introducing the concepts of statistical testing, we review the relevant statistical tests and compare them empirically in terms of false positive rate and statistical power as a function of the sample size (number of seeds) and effect size. We further investigate the robustness of these tests to violations of the most common hypotheses (normal distributions, same distributions, equal variances). Beside simulations, we compare empirical distributions obtained by running Soft-Actor Critic and Twin-Delayed Deep Deterministic Policy Gradient on Half-Cheetah. We conclude by providing guidelines and code to perform rigorous comparisons of RL algorithm performances.