mlsanityData to 7 Oct 2026

A Survey of Sim-to-Real Methods in RL: Progress, Prospects and Challenges with Foundation Models

arXiv 2502.13187 ↗First posted 18 Feb 2025cs.LG cs.AI cs.ROcited by 36

Not on the map. Why?

Abstract (arXiv)

Deep Reinforcement Learning (RL) has been explored and verified to be effective in solving decision-making tasks in various domains, such as robotics, transportation, recommender systems, etc. It learns from the interaction with environments and updates the policy using the collected experience. However, due to the limited real-world data and unbearable consequences of taking detrimental actions, the learning of RL policy is mainly restricted within the simulators. This practice guarantees safety in learning but introduces an inevitable sim-to-real gap in terms of deployment, thus causing degraded performance and risks in execution. There are attempts to solve the sim-to-real problems from different domains with various techniques, especially in the era with emerging techniques such as large foundations or language models that have cast light on the sim-to-real. This survey paper, to the best of our knowledge, is the first taxonomy that formally frames the sim-to-real techniques from key elements of the Markov Decision Process (State, Action, Transition, and Reward). Based on the framework, we cover comprehensive literature from the classic to the most advanced methods including the sim-to-real techniques empowered by foundation models, and we also discuss the specialties that are worth attention in different domains of sim-to-real problems. Then we summarize the formal evaluation process of sim-to-real performance with accessible code or benchmarks. The challenges and opportunities are also presented to encourage future exploration of this direction. We are actively maintaining a repository to include the most up-to-date sim-to-real research work to help domain researchers.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 115 of its 255 references that are in the map. First posted on arXiv 18 Feb 2025.

Builds on

115 of 255 references.

References · 115

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2017CARLA: An Open Urban Driving Simulatorcited by 2,431
  5. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  6. 2017Adversarial Discriminative Domain Adaptationcited by 1,309
  7. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  8. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 850
  9. 2019Solving Rubik's Cube with a Robot Handcited by 699
  10. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 698
  11. 2019RLBench: The Robot Learning Benchmark & Learning Environmentcited by 669
  12. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
38 more
  1. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  2. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
  3. 2017Safe Reinforcement Learning via Shieldingcited by 427
  4. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  5. 2020Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Surveycited by 388
  6. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  7. 2019Reinforcement Learning in Healthcare: A Surveycited by 308
  8. 2016Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networkscited by 307
  9. 2020A Brief Review of Domain Adaptationcited by 287
  10. 2019Challenges of Real-World Reinforcement Learningcited by 282
  11. 2023Orbit: A Unified Simulation Framework for Interactive Robot Learning Environmentscited by 239
  12. 2017Asymmetric Actor Critic for Image-Based Robot Learningcited by 202
  13. 2021TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviorscited by 192
  14. 2022Guided Conditional Diffusion for Controllable Traffic Simulationcited by 189
  15. 2021Reinforcement learning based recommender systems: A surveycited by 187
  16. 2022MIC: Masked Image Consistency for Context-Enhanced Domain Adaptationcited by 176
  17. 2018Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networkscited by 171
  18. 2019Action Robust Reinforcement Learning and Applications in Continuous Controlcited by 167
  19. 2019Active Domain Randomizationcited by 161
  20. 2023Waymax: An Accelerated, Data-Driven Simulator for Large-Scale Autonomous Driving Researchcited by 158
  21. 2024Accessing GPT-4 level Mathematical Olympiad Solutions via Monte Carlo Tree Self-refine with LLaMa-3 8Bcited by 156
  22. 2023PAC-NeRF: Physics Augmented Continuum Neural Radiance Fields for Geometry-Agnostic System Identificationcited by 145
  23. 2019CoLight: Learning Network-level Cooperation for Traffic Signal Controlcited by 133
  24. 2020SMARTS: Scalable Multi-Agent Reinforcement Learning Training School for Autonomous Drivingcited by 121
  25. 2020SoftGym: Benchmarking Deep Reinforcement Learning for Deformable Object Manipulationcited by 118
  26. 2019CityFlow: A Multi-Agent Reinforcement Learning Environment for Large Scale City Traffic Scenariocited by 114
  27. 2017Continuous Adaptation via Meta-Learning in Nonstationary and Competitive Environmentscited by 106
  28. 2021ManipulaTHOR: A Framework for Visual Object Manipulationcited by 101
  29. 2020Deep Reinforcement Learning for Intelligent Transportation Systems: A Surveycited by 94
  30. 2019RecSim: A Configurable Simulation Platform for Recommender Systemscited by 91
  31. 2020RL-CycleGAN: Reinforcement Learning Aware Simulation-To-Realcited by 89
  32. 2020Sim-to-Real Transfer for Vision-and-Language Navigationcited by 88
  33. 2018Virtual-Taobao: Virtualizing Real-world Online Retail Environment for Reinforcement Learningcited by 80
  34. 2022Sim-2-Sim Transfer for Vision-and-Language Navigation in Continuous Environmentscited by 80
  35. 2019A Survey on Traffic Signal Control Methodscited by 78
  36. 2020A Sim2Real Deep Learning Approach for the Transformation of Images from Multiple Vehicle-Mounted Cameras to a Semantically Segmented Image in Bird's Eye Viewcited by 71
  37. 2018Safe Reinforcement Learning with Model Uncertainty Estimatescited by 70
  38. 2021Understanding Domain Randomization for Sim-to-real Transfercited by 69

50 most cited of 115 All 115 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 13; 2026: 23.

36 papers.

Sort:

By area:

  1. 2025Uncertainty Quantification and Confidence Calibration in Large Language Models: A Surveycited by 93
  2. 2025WebSailor-V2: Bridging the Chasm to Proprietary Agents via Synthetic Data and Scalable Reinforcement Learningcited by 59
  3. 2025Residual Off-Policy RL for Finetuning Behavior Cloning Policiescited by 42
  4. 2025Scaling Agent Learning via Experience Synthesiscited by 40
  5. 2025SSRL: Self-Search Reinforcement Learningcited by 13
  6. 2025Generative AI in Transportation Planning: A Surveycited by 8
  7. 2025Linear Mixture Distributionally Robust Markov Decision Processescited by 5
  8. 2025DeepShade: Enable Shade Simulation by Text-conditioned Image Generationcited by 4
  9. 2026Conformal Feedback Alignment: Quantifying Answer-Level Reliability for Robust LLM Alignmentcited by 4
  10. 2026Data Pyramid for Embodied Manipulation: A Surveycited by 4
  11. 2025Diffusion-Based Forecasting for Uncertainty-Aware Model Predictive Controlcited by 2
  12. 2026Beware Untrusted Simulators -- Reward-Free Backdoor Attacks in Reinforcement Learningcited by 2
24 more
  1. 2025A Multi-Fidelity Control Variate Approach for Policy Gradient Estimationcited by 1
  2. 2025Learning human-to-robot handovers through 3D scene reconstructioncited by 1
  3. 2025Bi-Level Reinforcement Learning Pathway for Sim-to-Real Optimalitycited by 1
  4. 2026Sim-MSTNet: sim2real based Multi-task SpatioTemporal Network Traffic Forecastingcited by 1
  5. 2026Enhancing Goal Inference via Correction Timingcited by 1
  6. 2026Computational Pathology in the Era of Emerging Foundation and Agentic AI -- International Expert Perspectives on Clinical Integration and Translational Readinesscited by 1
  7. 2026The Sim-to-Real Gap of Foundation Model Agents: A Unified MDP Perspectivecited by 1
  8. 2026Tac-DINO: Learning Vision-Tactile Features with Patch Alignmentcited by 1
  9. 2026AndroidReality: How Far Are Mobile Agents from the Real World?cited by 1
  10. 2025Joint-Local Grounded Action Transformation for Sim-to-Real Transfer in Multi-Agent Traffic Controlcited by 0
  11. 2026Rationality Measurement and Theory for Reinforcement Learning Agentscited by 0
  12. 2026Online Adaptive Reinforcement Learning with Echo State Networks for Non-Stationary Dynamicscited by 0
  13. 2026Efficient Unsupervised Environment Design through Hierarchical Policy Representation Learningcited by 0
  14. 2026Developing AI Agents with Simulated Data: Why, what, and how?cited by 0
  15. 2026Automatic Generation of High-Performance RL Environmentscited by 0
  16. 2026Mitigating Data Scarcity in Spaceflight Applications for Offline Reinforcement Learning Using Physics-Informed Deep Generative Modelscited by 0
  17. 2026Application of Deep Reinforcement Learning to Event-Triggered Control for Networked Artificial Pancreas Systemscited by 0
  18. 2026Decoupled Delay Compensation: Enhancing Pre-trained MARL Policies via Learned Dynamics Filteringcited by 0
  19. 2026Transferable Reinforcement Learning via Probabilistic Latent Embeddings and Dynamic Policy Adaptation for Sim-to-Real Deploymentcited by 0
  20. 2026Targeting World Models to Compromise Robot Learning Pipelinescited by 0
  21. 2026IDEA: Insensitive to Dynamics Mismatch via Effect Alignment for Sim-to-Real Transfer in Multi-Agent Controlcited by 0
  22. 2026DADiff: Diffusion-Driven Cross-Domain Policy Adaptation for Reinforcement Learningcited by 0
  23. 2026Foundations of Reinforcement Learning and Control:Connections and New Perspectivescited by 0
  24. 2026Crossing the Cyber Divide: Sim-to-Sim and Sim-to-Real Transfer for RL Agentscited by 0