mlsanityData to 14 Oct 2026

Position: RL Researchers Need to Distinguish Between Solving Simulators and Using Simulators as a Proxy

arXiv 2606.28433 ↗First posted 26 Jun 2026cs.LGcited by 1

On the map: Deep reinforcement learning methods → RL for robot locomotion and manipulation

Abstract (arXiv)

One goal in reinforcement learning (RL) research is to understand general-purpose sequential decision-making, using benchmark simulators as a proxy for learning in deployment settings. When running experiments, however, the goal of achieving high performance in the simulator can mutate into focusing exclusively on solving the simulator. To achieve high scores, researchers may adopt solutions exclusively meant for solving simulators, rather than learning while the agent is deployed outside a simulator. Solving simulators is also worthy of investigation, but it is a fundamentally different RL research question. In this paper, we argue that RL researchers need to distinguish between two use cases of simulators: solving simulators and using simulators as a proxy for learning in deployment. We first discuss how these two use-cases are importantly different, in terms of constraints on how the agent can use the simulator, which algorithms are appropriate, and which evaluation metrics are appropriate. We then highlight several issues and misleading conclusions that can occur by not making the distinction between these two settings clear, supported with examples and simple experiments. This work is a call to the community to begin clearly distinguishing how they are using simulators in their work, hopefully sparking further discussion on which empirical practices work best in each setting.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods. Builds on 34 of its 60 references that are in the map. First posted on arXiv 26 Jun 2026.

Builds on

34 of 60 references.

References · 34

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Trust Region Policy Optimizationcited by 2,580
  4. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  5. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
  6. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  7. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  8. 2020Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Surveycited by 413
  9. 2020dm_control: Software and Tasks for Continuous Controlcited by 375
  10. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 371
  11. 2019End-to-End Safe Reinforcement Learning through Barrier Functions for Safety-Critical Continuous Control Taskscited by 289
  12. 2017DeepStack: Expert-Level Artificial Intelligence in No-Limit Pokercited by 233
22 more
  1. 2019MineRL: A Large-Scale Dataset of Minecraft Demonstrationscited by 215
  2. 2019Go-Explore: a New Approach for Hard-Exploration Problemscited by 212
  3. 2019Behaviour Suite for Reinforcement Learningcited by 132
  4. 2019Learning with Good Feature Representations in Bandits and in RL with a Generative Modelcited by 127
  5. 2022Discovered Policy Optimisationcited by 104
  6. 2024Simplifying Deep Temporal Difference Learningcited by 78
  7. 2018Benchmarking Reinforcement Learning Algorithms on Real-World Robotscited by 65
  8. 2018A Contextual Bandit Bake-offcited by 64
  9. 2022EnvPool: A Highly Parallel Reinforcement Learning Environment Execution Enginecited by 62
  10. 2023Hyperparameters in Reinforcement Learning and How To Tune Themcited by 59
  11. 2023Jumanji: a Diverse Suite of Scalable Reinforcement Learning Environments in JAXcited by 58
  12. 2023Empirical Design in Reinforcement Learningcited by 55
  13. 2018ViZDoom Competitions: Playing Doom from Pixelscited by 51
  14. 2020Learning to Play Table Tennis From Scratch using Muscular Robotscited by 35
  15. 2022Controlling Commercial Cooling Systems Using Reinforcement Learningcited by 29
  16. 2024Application-Driven Innovation in Machine Learningcited by 24
  17. 2024A Method for Evaluating Hyperparameter Sensitivity in Reinforcement Learningcited by 15
  18. 2024The Power of Resets in Online Reinforcement Learningcited by 14
  19. 2022The Impact of Task Underspecification in Evaluating Deep Reinforcement Learningcited by 10
  20. 2023Materials Discovery with Extreme Properties via Reinforcement Learning-Guided Combinatorial Chemistrycited by 5
  21. 2023Transfer learning for process design with reinforcement learningcited by 4
  22. 2025The Formalism-Implementation Gap in Reinforcement Learning Researchcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Sparks of In Silico Cognitive Science: Theories from Simulated Data Can Generalize to Humanscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Overcoming the Sim-to-Real Gap: Leveraging Simulation to Learn to Explore for Real-World RL
  2. 2026Reinforcement Learning in the Real World: A Survey of Statistical Challenges and Future Directions
  3. 2024Benchmarks for Reinforcement Learning with Biased Offline Data and Imperfect Simulators
  4. 2025A Survey of Sim-to-Real Methods in RL: Progress, Prospects and Challenges with Foundation Models
  5. 2025Performance Comparisons of Reinforcement Learning Algorithms for Sequential Experimental Design
  6. 2024A Review of Nine Physics Engines for Reinforcement Learning Research
  7. 2026Reinforcement Learning: From Algorithms To Foundation Models
  8. 2026Reinforcement Learning in Operational Research: A Technical Review and Practical Roadmap