mlsanityData to 14 Oct 2026

Position: Benchmarking is Limited in Reinforcement Learning Research

arXiv 2406.16241 ↗First posted 23 Jun 2024cs.LG stat.MEcited by 15

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Novel reinforcement learning algorithms, or improvements on existing ones, are commonly justified by evaluating their performance on benchmark environments and are compared to an ever-changing set of standard algorithms. However, despite numerous calls for improvements, experimental practices continue to produce misleading or unsupported claims. One reason for the ongoing substandard practices is that conducting rigorous benchmarking experiments requires substantial computational time. This work investigates the sources of increased computation costs in rigorous experiment designs. We show that conducting rigorous performance benchmarks will likely have computational costs that are often prohibitive. As a result, we argue for using an additional experimentation paradigm to overcome the limitations of benchmarking.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 19 of its 49 references that are in the map. First posted on arXiv 23 Jun 2024.

Builds on

19 of 49 references.

References · 19

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  6. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  7. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  8. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  9. 2017Deep Reinforcement Learning that Matterscited by 754
  10. 2019Solving Rubik's Cube with a Robot Handcited by 734
  11. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  12. 2020First return, then explorecited by 266
7 more
  1. 2017Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Controlcited by 93
  2. 2018How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experimentscited by 60
  3. 2023Hyperparameters in Reinforcement Learning and How To Tune Themcited by 59
  4. 2023Empirical Design in Reinforcement Learningcited by 55
  5. 2018The Mirage of Action-Dependent Baselines in Reinforcement Learningcited by 48
  6. 2018Re-evaluating Evaluationcited by 47
  7. 2020Evaluating the Performance of Reinforcement Learning Algorithmscited by 36

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 9; 2026: 4.

15 papers.

Sort:

By area:

  1. 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 93
  2. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 15
  3. 2025Benchmarking Quantum Reinforcement Learningcited by 10
  4. 2025There are no Champions in Supervised Long-Term Time Series Forecastingcited by 8
  5. 2024Craftium: Bridging Flexibility and Efficiency for Rich 3D Single- and Multi-Agent Environmentscited by 4
  6. 2025Investigating Memory in Model-Free RL with POPGym Arcadecited by 3
  7. 2025The Formalism-Implementation Gap in Reinforcement Learning Researchcited by 3
  8. 2025gfnx: Fast and Scalable Library for Generative Flow Networks in JAXcited by 3
  9. 2025Humans Coexist, So Must Embodied Artificial Agentscited by 2
  10. 2026Don't Claim Benchmark-Oriented Optimization Improves General Coding Capability -- Diverse Evaluation Is Requiredcited by 2
  11. 2026Behavior-Consistent Deep Reinforcement Learningcited by 1
  12. 2024A Research Agenda for Usability and Generalisation in Reinforcement Learningcited by 0
3 more
  1. 2025Minding Motivation: The Effect of Intrinsic Motivation on Agent Behaviorscited by 0
  2. 2026Beyond One-Size-Fits-All: Diagnosis-Driven Online Reinforcement Learning with Offline Priorscited by 0
  3. 2026Jaxolotl: A Unified High-Performance Benchmark Suite for LTL-Based Multi-Task RLcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Reliable validation of Reinforcement Learning Benchmarks
  2. 2024Can we hop in general? A discussion of benchmark selection and design using the Hopper environment
  3. 2024Open RL Benchmark: Comprehensive Tracked Experiments for Reinforcement Learning
  4. 2025Performance Comparisons of Reinforcement Learning Algorithms for Sequential Experimental Design
  5. 2026Benchmarking Reinforcement Learning via Stochastic Converse Optimality: Generating Systems with Known Optimal Policies
  6. 2025Benchmarking Partial Observability in Reinforcement Learning with a Suite of Memory-Improvable Domains
  7. 2024A Method for Evaluating Hyperparameter Sensitivity in Reinforcement Learning
  8. 2024Revisiting Safe Exploration in Safe Reinforcement learning