mlsanityData to 14 Oct 2026

Overcoming the Sim-to-Real Gap: Leveraging Simulation to Learn to Explore for Real-World RL

arXiv 2410.20254 ↗First posted 26 Oct 2024cs.LG cs.RO stat.MLcited by 21

On the map: Deep reinforcement learning methods → RL for robot locomotion and manipulation

Abstract (arXiv)

In order to mitigate the sample complexity of real-world reinforcement learning, common practice is to first train a policy in a simulator where samples are cheap, and then deploy this policy in the real world, with the hope that it generalizes effectively. Such direct sim2real transfer is not guaranteed to succeed, however, and in cases where it fails, it is unclear how to best utilize the simulator. In this work, we show that in many regimes, while direct sim2real transfer may fail, we can utilize the simulator to learn a set of exploratory policies which enable efficient exploration in the real world. In particular, in the setting of low-rank MDPs, we show that coupling these exploratory policies with simple, practical approaches -- least-squares regression oracles and naive randomized exploration -- yields a polynomial sample complexity in the real world, an exponential improvement over direct sim2real transfer, or learning without access to a simulator. To the best of our knowledge, this is the first evidence that simulation transfer yields a provable gain in reinforcement learning in settings where direct sim2real transfer fails. We validate our theoretical results on several realistic robotic simulators and a real-world robotic sim2real task, demonstrating that transferring exploratory policies can yield substantial gains in practice as well.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods. Builds on 48 of its 77 references that are in the map. First posted on arXiv 26 Oct 2024.

Builds on

48 of 77 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024ASID: Active Exploration for System Identification in Robotic Manipulation18 Apr

Other references · 47

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  4. 2018Exploration by Random Network Distillationcited by 993
  5. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  6. 2019Solving Rubik's Cube with a Robot Handcited by 734
  7. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  8. 2019Provably Efficient Reinforcement Learning with Linear Function Approximationcited by 520
  9. 2021RMA: Rapid Motor Adaptation for Legged Robotscited by 508
  10. 2016Learning to reinforcement learncited by 377
  11. 2020Learning Agile Robotic Locomotion Skills by Imitating Animalscited by 251
  12. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 233
35 more
  1. 2019Efficient Exploration via State Marginal Matchingcited by 188
  2. 2021Maximum Entropy RL (Provably) Solves Some Robust RL Problemscited by 176
  3. 2018Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networkscited by 174
  4. 2019Active Domain Randomizationcited by 170
  5. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 161
  6. 2020SUNRISE: A Simple Unified Framework for Ensemble Learning in Deep Reinforcement Learningcited by 152
  7. 2021Representation Learning for Online and Offline RL in Low-rank MDPscited by 129
  8. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 86
  9. 2020One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RLcited by 79
  10. 2020Provably Efficient Reward-Agnostic Navigation with Linear Value Iterationcited by 56
  11. 2022Making Linear MDPs Practical via Contrastive Representation Learningcited by 52
  12. 2022Reward-Free RL is No Harder Than Reward-Aware RL in Linear Markov Decision Processescited by 49
  13. 2022Provable Benefits of Representational Transfer in Reinforcement Learningcited by 38
  14. 2022Instance-Dependent Near-Optimal Policy Identification in Linear MDPs via Online Experiment Designcited by 33
  15. 2022Guarantees for Epsilon-Greedy Reinforcement Learning with Function Approximationcited by 31
  16. 2023On the Importance of Exploration for Generalization in Reinforcement Learningcited by 31
  17. 2021On the Power of Multitask Representation Learning in Linear MDPcited by 30
  18. 2022Provable Benefit of Multitask Representation Learning in Reinforcement Learningcited by 29
  19. 2021Sample-Efficient Reinforcement Learning Is Feasible for Linearly Realizable MDPs with Limited Revisitingcited by 28
  20. 2020Never Stop Learning: The Effectiveness of Fine-Tuning in Robotic Reinforcement Learningcited by 23
  21. 2021Efficient Local Planning with Linear Function Approximationcited by 23
  22. 2021When Is Generalizable Reinforcement Learning Tractable?cited by 20
  23. 2021Online Target Q-learning with Reverse Experience Replay: Efficiently finding the Optimal Policy for Linear MDPscited by 19
  24. 2021On Query-efficient Planning in MDPs under Linear Realizability of the Optimal State-value Functioncited by 18
  25. 2022Transfer RL across Observation Feature Spaces via Model-Based Regularizationcited by 18
  26. 2023Optimal Exploration for Model-Based RL in Nonlinear Systemscited by 18
  27. 2023Searching for High-Value Molecules Using Reinforcement Learning and Transformerscited by 18
  28. 2023Efficient Model-Free Exploration in Low-Rank MDPscited by 16
  29. 2021A Fully Problem-Dependent Regret Lower Bound for Finite-Horizon MDPscited by 15
  30. 2024The Power of Resets in Online Reinforcement Learningcited by 14
  31. 2023Cherry-Picking with Reinforcement Learning : Robust Dynamic Grasping in Unstable Conditionscited by 11
  32. 2020Provably Efficient Model-based Policy Adaptationcited by 8
  33. 2022A Few Expert Queries Suffices for Sample-Efficient RL with Resets and Linear Value Approximationcited by 7
  34. 2022On the Power of Pre-training for Generalization in RL: Provable Benefits and Hardnesscited by 7
  35. 2022Provably Sample-Efficient RL with Side Information about Latent Dynamicscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 17.

21 papers.

Sort:

By area:

  1. 2025A Survey of Sim-to-Real Methods in RL: Progress, Prospects and Challenges with Foundation Modelscited by 38
  2. 2026Simulation Distillation: Pretraining World Models in Simulation for Rapid Real-World Adaptationcited by 6
  3. 2025SimLauncher: Launching Sample-Efficient Real-world Robotic Reinforcement Learning via Simulation Pre-trainingcited by 5
  4. 2026Learning to Act While Waiting: RL Finetuning of Generalist Robot Policies Under Inference Latencycited by 2
  5. 2025Find the Fruit: Zero-Shot Sim2Real RL for Occlusion-Aware Plant Manipulationcited by 1
  6. 2025Blindfolded Experts Generalize Better: Insights from Robotic Manipulation and Videogamescited by 1
  7. 2026OOM-RL: Out-of-Money Reinforcement Learning Market-Driven Alignment for LLM-Based Multi-Agent Systemscited by 1
  8. 2026Data-driven robust Markov decision processes on Borel spaces: performance guarantees via an axiomatic approachcited by 0
  9. 2026Escaping Offline Pessimism: Vector-Field Reward Shaping for Safe Frontier Explorationcited by 0
  10. 2026TacSIm: A Dataset and Benchmark for Football Tactical Style Imitationcited by 0
  11. 2026Learning Task-Invariant Properties via Dreamer: Enabling Efficient Policy Transfer for Quadruped Robotscited by 0
  12. 2026Enhancing RL Generalizability in Robotics through SHAP Analysis of Algorithms and Hyperparameterscited by 0
9 more
  1. 2026When Does Hierarchy Help? Benchmarking Agent Coordination in Event-Driven Industrial Schedulingcited by 0
  2. 2026Mind the Sim-to-Real Gap & Think Like a Scientistcited by 0
  3. 2026Zero-shot Transfer of Reinforcement Learning Control Policies for the Swing-Up and Stabilization of a Cart-Pole Systemcited by 0
  4. 2026Beyond One-Size-Fits-All: Diagnosis-Driven Online Reinforcement Learning with Offline Priorscited by 0
  5. 2026Anticipatory Reinforcement Learning for Trajectory Trackingcited by 0
  6. 2026Measure the Sim-to-Real Gap: Designing an Affordable Real-World Benchmark Platform for Reinforcement Learning in AIoT Systemscited by 0
  7. 2026Reinforcement Learning for the Full Strawberry Harvesting Process: Obstacle Separation, Detachment, and Placementcited by 0
  8. 2026Provably Safe Sim-to-Real Transfercited by 0
  9. 2026A Cyber Range Evaluation of Autonomous Network Incident Response Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Revealing the Challenges of Sim-to-Real Transfer in Model-Based Reinforcement Learning via Latent Space Modeling
  2. 2026Abstract Sim2Real through Approximate Information States
  3. 2022When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learning
  4. 2020Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey
  5. 2026Position: RL Researchers Need to Distinguish Between Solving Simulators and Using Simulators as a Proxy
  6. 2023H2O+: An Improved Framework for Hybrid Offline-and-Online RL with Dynamics Gaps
  7. 2026Theoretical Foundations and Effective Algorithms for Policy-Aware Simulator Learning
  8. 2026A Balanced Data Diet: Addressing the Exploration Bottleneck in Mega-Scale RL for Robot Control