mlsanityData to 14 Oct 2026

The Formalism-Implementation Gap in Reinforcement Learning Research

arXiv 2510.16175 ↗First posted 17 Oct 2025cs.LG cs.AIcited by 3

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

The last decade has seen an upswing in interest and adoption of reinforcement learning (RL) techniques, in large part due to its demonstrated capabilities at performing certain tasks at "super-human levels". This has incentivized the community to prioritize research that demonstrates RL agent performance, often at the expense of research aimed at understanding their learning dynamics. Performance-focused research runs the risk of overfitting on academic benchmarks -- thereby rendering them less useful -- which can make it difficult to transfer proposed techniques to novel problems. Further, it implicitly diminishes work that does not push the performance-frontier, but aims at improving our understanding of these techniques. This paper argues two points: (i) RL research should stop focusing solely on demonstrating agent capabilities, and focus more on advancing the science and understanding of reinforcement learning; and (ii) we need to be more precise on how our benchmarks map to the underlying mathematical formalisms. We use the popular Arcade Learning Environment (ALE; Bellemare et al., 2013) as an example of a benchmark that, despite being increasingly considered "saturated", can be effectively used for developing this understanding, and facilitating the deployment of RL techniques in impactful real-world problems.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 30 of its 50 references that are in the map. First posted on arXiv 17 Oct 2025.

Builds on

30 of 50 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024A Method for Evaluating Hyperparameter Sensitivity in Reinforcement Learning10 Dec
  2. 2024Three Dogmas of Reinforcement Learning15 Jul
  3. 2024On the consistency of hyper-parameter selection in value-based deep reinforcement learning25 Jun
  4. 2024Position: Benchmarking is Limited in Reinforcement Learning Research23 Jun

Other references · 26

  1. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  2. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  3. 2017Deep Reinforcement Learning that Matterscited by 754
  4. 2019Model-Based Reinforcement Learning for Ataricited by 502
  5. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
  6. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
  7. 2018Dopamine: A Research Framework for Deep Reinforcement Learningcited by 122
  8. 2018Generalization and Regularization in DQNcited by 118
  9. 2019When to use parametric models in reinforcement learning?cited by 118
  10. 2020Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Researchcited by 78
  11. 2022The State of Sparse Training in Deep Reinforcement Learningcited by 61
  12. 2019Hyperbolic Discounting and Learning over Multiple Horizonscited by 60
14 more
  1. 2023Empirical Design in Reinforcement Learningcited by 55
  2. 2022Atari-5: Distilling the Arcade Learning Environment down to Five Gamescited by 44
  3. 2023Proto-Value Networks: Scaling Representation Learning with Auxiliary Taskscited by 42
  4. 2023Small batch deep reinforcement learningcited by 36
  5. 2023OCAtari: Object-Centric Atari 2600 Reinforcement Learning Environmentscited by 30
  6. 2019Discounted Reinforcement Learning Is Not an Optimization Problemcited by 26
  7. 2024Position: Why We Must Rethink Empirical Research in Machine Learningcited by 19
  8. 2019Separating value functions across time-scalescited by 15
  9. 2022Deep Hedging: Continuous Reinforcement Learning for Hedging of General Portfolios across Multiple Risk Aversionscited by 13
  10. 2023A Better Match for Drivers and Riders: Reinforcement Learning at Lyftcited by 9
  11. 2020Multiplayer Support for the Arcade Learning Environmentcited by 8
  12. 2020Reinforcement Learning Agents for Ubisoft's Roller Championscited by 6
  13. 2022Mask Atari for Deep Reinforcement Learning as POMDP Benchmarkscited by 5
  14. 2024CALE: Continuous Arcade Learning Environmentcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Controllability in preference-conditioned multi-objective reinforcement learningcited by 1
  2. 2026Position: RL Researchers Need to Distinguish Between Solving Simulators and Using Simulators as a Proxycited by 1
  3. 2026surprisal is Not a Theorycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Reliable validation of Reinforcement Learning Benchmarks
  2. 2024Can we hop in general? A discussion of benchmark selection and design using the Hopper environment
  3. 2024Open RL Benchmark: Comprehensive Tracked Experiments for Reinforcement Learning
  4. 2023Improving and Benchmarking Offline Reinforcement Learning Algorithms
  5. 2025NeoRL-2: Near Real-World Benchmarks for Offline Reinforcement Learning with Extended Realistic Scenarios
  6. 2025Revealing the Challenges of Sim-to-Real Transfer in Model-Based Reinforcement Learning via Latent Space Modeling
  7. 2024A Review of Nine Physics Engines for Reinforcement Learning Research
  8. 2026Reinforcement Learning in the Real World: A Survey of Statistical Challenges and Future Directions