mlsanityData to 14 Oct 2026

Can we hop in general? A discussion of benchmark selection and design using the Hopper environment

arXiv 2410.08870 ↗First posted 11 Oct 2024cs.LGcited by 7

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Empirical, benchmark-driven testing is a fundamental paradigm in the current RL community. While using off-the-shelf benchmarks in reinforcement learning (RL) research is a common practice, this choice is rarely discussed. Benchmark choices are often done based on intuitive ideas like "legged robots" or "visual observations". In this paper, we argue that benchmarking in RL needs to be treated as a scientific discipline itself. To illustrate our point, we present a case study on different variants of the Hopper environment to show that the selection of standard benchmarking suites can drastically change how we judge performance of algorithms. The field does not have a cohesive notion of what the different Hopper environments are representative - they do not even seem to be representative of each other. Our experimental results suggests a larger issue in the deep RL literature: benchmark choices are neither commonly justified, nor does there exist a language that could be used to justify the selection of certain environments. This paper concludes with a discussion of the requirements for proper discussion and evaluations of benchmarks and recommends steps to start a dialogue towards this goal.

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 46 of its 68 references that are in the map. First posted on arXiv 11 Oct 2024.

Builds on

46 of 68 references.

References · 46

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2015Continuous control with deep reinforcement learningcited by 3,407
  3. 2015Trust Region Policy Optimizationcited by 2,580
  4. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  5. 2023LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learningcited by 1,410
  6. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  7. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  8. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 773
  9. 2017Deep Reinforcement Learning that Matterscited by 754
  10. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  11. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 689
  12. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 520
34 more
  1. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  2. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 511
  3. 2017Minimax Regret Bounds for Reinforcement Learningcited by 503
  4. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 483
  5. 2020dm_control: Software and Tasks for Continuous Controlcited by 375
  6. 2021AI and the Everything in the Whole Wide World Benchmarkcited by 349
  7. 2018TextWorld: A Learning Environment for Text-based Gamescited by 310
  8. 2017StarCraft II: A New Challenge for Reinforcement Learningcited by 274
  9. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
  10. 2019Google Research Football: A Novel Reinforcement Learning Environmentcited by 260
  11. 2021Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithmscited by 223
  12. 2020The NetHack Learning Environmentcited by 198
  13. 2021ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrationscited by 197
  14. 2019Tighter Problem-Dependent Regret Bounds in Reinforcement Learning without Domain Knowledge using Value Function Boundscited by 195
  15. 2019OpenSpiel: A Framework for Reinforcement Learning in Gamescited by 186
  16. 2016ViZDoom: A Doom-based AI Research Platform for Visual Reinforcement Learningcited by 179
  17. 2020Reward-Free Exploration for Reinforcement Learningcited by 177
  18. 2019Is a Good Representation Sufficient for Sample Efficient Reinforcement Learning?cited by 140
  19. 2019Behaviour Suite for Reinforcement Learningcited by 132
  20. 2021Continual World: A Robotic Benchmark For Continual Reinforcement Learningcited by 116
  21. 2020CausalWorld: A Robotic Manipulation Benchmark for Causal Structure and Transfer Learningcited by 109
  22. 2015Sample Complexity of Episodic Fixed-Horizon Reinforcement Learningcited by 106
  23. 2020Episodic Reinforcement Learning in Finite MDPs: Minimax Lower Bounds Revisitedcited by 106
  24. 2021Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Developmentcited by 85
  25. 2021FinRL: Deep Reinforcement Learning Framework to Automate Trading in Quantitative Financecited by 58
  26. 2023Empirical Design in Reinforcement Learningcited by 55
  27. 2021safe-control-gym: a Unified Benchmark Suite for Safe Learning-based Control and Reinforcement Learning in Roboticscited by 50
  28. 2016On Lower Bounds for Regret in Reinforcement Learningcited by 40
  29. 2023Bridging RL Theory and Practice with the Effective Horizoncited by 31
  30. 2021DACBench: A Benchmark Library for Dynamic Algorithm Configurationcited by 28
  31. 2022Simplifying Model-based RL: Learning Representations, Latent-space Models, and Policies with One Objectivecited by 28
  32. 2022CompoSuite: A Compositional Reinforcement Learning Benchmarkcited by 21
  33. 2017Benchmark Environments for Multitask Learning in Continuous Domainscited by 14
  34. 2024Exploration is Harder than Prediction: Cryptographically Separating Reinforcement Learning from Supervised Learningcited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 3; 2026: 2.

7 papers.

  1. 2025Relative Entropy Pathwise Policy Optimizationcited by 7
  2. 2025Replicable Reinforcement Learning with Linear Function Approximationcited by 7
  3. 2024Stealing That Free Lunch: Exposing the Limits of Dyna-Style Reinforcement Learningcited by 4
  4. 2026Behavior-Consistent Deep Reinforcement Learningcited by 1
  5. 2026Position: Behavioral Systems Require Behavioral Testscited by 1
  6. 2024A Research Agenda for Usability and Generalisation in Reinforcement Learningcited by 0
  7. 2025A Forensic Analysis of Synthetic Data in RL: Diagnosing and Solving Algorithmic Failures in Model-Based Policy Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Open RL Benchmark: Comprehensive Tracked Experiments for Reinforcement Learning
  2. 2022Reliable validation of Reinforcement Learning Benchmarks
  3. 2024Position: Benchmarking is Limited in Reinforcement Learning Research
  4. 2025Gym4ReaL: A Suite for Benchmarking Real-World Reinforcement Learning
  5. 2025Rethinking RL Evaluation: Can Benchmarks Truly Reveal Failures of RL Methods?
  6. 2026Benchmarking Reinforcement Learning via Stochastic Converse Optimality: Generating Systems with Known Optimal Policies
  7. 2024ARLBench: Flexible and Efficient Benchmarking for Hyperparameter Optimization in Reinforcement Learning
  8. 2025Benchmarking Partial Observability in Reinforcement Learning with a Suite of Memory-Improvable Domains