mlsanityData to 7 Oct 2026

Deep Reinforcement Learning at the Edge of the Statistical Precipice

arXiv 2108.13264 ↗First posted 30 Aug 2021cs.LG cs.AI stat.MEcited by 781

Not on the map. Why?

Cited by 107 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Deep reinforcement learning (RL) algorithms are predominantly evaluated by comparing their relative performance on a large suite of tasks. Most published results on deep RL benchmarks compare point estimates of aggregate performance such as mean and median scores across tasks, ignoring the statistical uncertainty implied by the use of a finite number of training runs. Beginning with the Arcade Learning Environment (ALE), the shift towards computationally-demanding benchmarks has led to the practice of evaluating only a small number of runs per task, exacerbating the statistical uncertainty in point estimates. In this paper, we argue that reliable evaluation in the few run deep RL regime cannot ignore the uncertainty in results without running the risk of slowing down progress in the field. We illustrate this point using a case study on the Atari 100k benchmark, where we find substantial discrepancies between conclusions drawn from point estimates alone versus a more thorough statistical analysis. With the aim of increasing the field's confidence in reported results with a handful of runs, we advocate for reporting interval estimates of aggregate performance and propose performance profiles to account for the variability in results, as well as present more robust and efficient aggregate metrics, such as interquartile mean scores, to achieve small uncertainty in results. Using such statistical tools, we scrutinize performance evaluations of existing algorithms on other widely used RL benchmarks including the ALE, Procgen, and the DeepMind Control Suite, again revealing discrepancies in prior comparisons. Our findings call for a change in how we evaluate performance in deep RL, for which we present a more rigorous evaluation methodology, accompanied with an open-source library rliable, to prevent unreliable results from stagnating the field.

Key facts. Cited by 781 papers in the map (data through 7 Oct 2026). Builds on 74 of its 121 references that are in the map. First posted on arXiv 30 Aug 2021.

Builds on

74 of 121 references.

References · 74

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  5. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  6. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  7. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  8. 2020Mastering Atari with Discrete World Modelscited by 990
  9. 2018DeepMind Control Suitecited by 935
  10. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  11. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  12. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
38 more
  1. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  2. 2017Deep Reinforcement Learning that Matterscited by 727
  3. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  4. 2017Evolution Strategies as a Scalable Alternative to Reinforcement Learningcited by 555
  5. 2019Model-Based Reinforcement Learning for Ataricited by 495
  6. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  7. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  8. 2020Reinforcement Learning with Augmented Datacited by 452
  9. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  10. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 377
  11. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  12. 2020Improving Reproducibility in Machine Learning Research (A Report from the NeurIPS 2019 Reproducibility Program)cited by 318
  13. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
  14. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 300
  15. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
  16. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
  17. 2017Are GANs Created Equal? A Large-Scale Studycited by 267
  18. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 228
  19. 2019Go-Explore: a New Approach for Hard-Exploration Problemscited by 207
  20. 2021Accounting for Variance in Machine Learning Benchmarkscited by 195
  21. 2020Prioritized Level Replaycited by 190
  22. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 173
  23. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 150
  24. 2021The Benchmark Lotterycited by 133
  25. 2019Fast Task Inference with Variational Intrinsic Successor Featurescited by 131
  26. 2018Dopamine: A Research Framework for Deep Reinforcement Learningcited by 121
  27. 2020Phasic Policy Gradientcited by 121
  28. 2019When to use parametric models in reinforcement learning?cited by 118
  29. 2019Unsupervised Learning of Object Keypoints for Perception and Controlcited by 109
  30. 2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 107
  31. 2017On the State of the Art of Evaluation in Neural Language Modelscited by 103
  32. 2017Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Controlcited by 92
  33. 2021The MultiBERTs: BERT Reproductions for Robustness Analysiscited by 89
  34. 2020Improving Generalization in Reinforcement Learning with Mixture Regularizationcited by 81
  35. 2021Decoupling Value and Policy for Generalization in Reinforcement Learningcited by 81
  36. 2020Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Researchcited by 78
  37. 2018Simple random search provides a competitive approach to reinforcement learningcited by 75
  38. 2017Reporting Score Distributions Makes a Difference: Performance Study of LSTM-networks for Sequence Taggingcited by 71

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 18; 2022: 116; 2023: 137; 2024: 175; 2025: 141; 2026: 194.

781 papers.

Sort:

By area:

  1. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  2. 2023Eureka: Human-Level Reward Design via Coding Large Language Modelscited by 542
  3. 2022Temporal Difference Learning for Model Predictive Controlcited by 375
  4. 2022Transformers are Sample-Efficient World Modelscited by 317
  5. 2024Diffusion for World Modeling: Visual Details Matter in Ataricited by 285
  6. 2022Masked Visual Pre-training for Motor Controlcited by 246
  7. 2022The Primacy Bias in Deep Reinforcement Learningcited by 242
  8. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 236
  9. 2021Mastering Atari Games with Limited Datacited by 235
  10. 2022Multi-Game Decision Transformerscited by 212
  11. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 185
  12. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 171
38 more
  1. 2022Evolving Curricula with Regret-Based Environment Designcited by 166
  2. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 145
  3. 2023Transformer-based World Models Are Happy With 100k Interactionscited by 137
  4. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 136
  5. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 136
  6. 2024iVideoGPT: Interactive VideoGPTs are Scalable World Modelscited by 136
  7. 2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
  8. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
  9. 2023GEO-Bench: Toward Foundation Models for Earth Monitoringcited by 122
  10. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 121
  11. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Librarycited by 117
  12. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 114
  13. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learningcited by 112
  14. 2024BetterBench: Assessing AI Benchmarks, Uncovering Issues, and Establishing Best Practicescited by 112
  15. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
  16. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  17. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous controlcited by 104
  18. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RLcited by 103
  19. 2023Synthetic Experience Replaycited by 100
  20. 2024TabDPT: Scaling Tabular Foundation Models on Real Datacited by 97
  21. 2022Design-Bench: Benchmarks for Data-Driven Offline Model-Based Optimizationcited by 96
  22. 2023A Survey on Fairness in Large Language Modelscited by 91
  23. 2025The Art of Scaling Reinforcement Learning Compute for LLMscited by 91
  24. 2022Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RLcited by 89
  25. 2024Foundation Policies with Hilbert Representationscited by 89
  26. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 89
  27. 2023Large Language Models as Generalizable Policies for Embodied Taskscited by 86
  28. 2023Bootstrap Your Own Skills: Learning to Solve New Tasks with Large Language Model Guidancecited by 85
  29. 2025Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluationcited by 83
  30. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  31. 2022Can Wikipedia Help Offline Reinforcement Learning?cited by 76
  32. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RLcited by 75
  33. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizescited by 72
  34. 2023Deep Reinforcement Learning with Plasticity Injectioncited by 71
  35. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 69
  36. 2021DR3: Value-Based Deep Reinforcement Learning Requires Explicit Regularizationcited by 66
  37. 2022CIC: Contrastive Intrinsic Control for Unsupervised Skill Discoverycited by 66
  38. 2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 66

50 most cited of 781 All 781 (JSON)