# A Survey of Sim-to-Real Methods in RL: Progress, Prospects and Challenges with Foundation Models

arXiv: https://arxiv.org/abs/2502.13187 · HTML: https://mlsanity.com/paper/2502.13187/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods.
- Builds on 115 of its 255 references that are in the map.
- First posted on arXiv 18 Feb 2025.

## Placed in

- Deep reinforcement learning methods / [RL for robot locomotion and manipulation](https://mlsanity.com/deeprl/rl-for-robot-locomotion-and-manipulation/)

## Builds on (most cited first, 30 of 115)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [CARLA: An Open Urban Driving Simulator](https://mlsanity.com/paper/1711.03938/) (2017)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Adversarial Discriminative Domain Adaptation](https://mlsanity.com/paper/1702.05464/) (2017)
- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [Deep Reinforcement Learning for Autonomous Driving: A Survey](https://mlsanity.com/paper/2002.00444/) (2020)
- [RLBench: The Robot Learning Benchmark & Learning Environment](https://mlsanity.com/paper/1909.12271/) (2019)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Tasks](https://mlsanity.com/paper/2112.03227/) (2021)
- [robosuite: A Modular Simulation Framework and Benchmark for Robot Learning](https://mlsanity.com/paper/2009.12293/) (2020)
- [Safe Reinforcement Learning via Shielding](https://mlsanity.com/paper/1708.08611/) (2017)
- [Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey](https://mlsanity.com/paper/2009.13303/) (2020)
- [Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment](https://mlsanity.com/paper/2308.05374/) (2023)
- [dm_control: Software and Tasks for Continuous Control](https://mlsanity.com/paper/2006.12983/) (2020)
- [Reinforcement Learning in Healthcare: A Survey](https://mlsanity.com/paper/1908.08796/) (2019)
- [Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks](https://mlsanity.com/paper/1612.05424/) (2016)
- [Orbit: A Unified Simulation Framework for Interactive Robot Learning Environments](https://mlsanity.com/paper/2301.04195/) (2023)
- [Challenges of Real-World Reinforcement Learning](https://mlsanity.com/paper/1904.12901/) (2019)
- [A Brief Review of Domain Adaptation](https://mlsanity.com/paper/2010.03978/) (2020)
- [Asymmetric Actor Critic for Image-Based Robot Learning](https://mlsanity.com/paper/1710.06542/) (2017)
- [TrafficSim: Learning to Simulate Realistic Multi-Agent Behaviors](https://mlsanity.com/paper/2101.06557/) (2021)
- [Guided Conditional Diffusion for Controllable Traffic Simulation](https://mlsanity.com/paper/2210.17366/) (2022)
- [Reinforcement learning based recommender systems: A survey](https://mlsanity.com/paper/2101.06286/) (2021)
- [MIC: Masked Image Consistency for Context-Enhanced Domain Adaptation](https://mlsanity.com/paper/2212.01322/) (2022)
- [Sim-to-Real via Sim-to-Sim: Data-efficient Robotic Grasping via Randomized-to-Canonical Adaptation Networks](https://mlsanity.com/paper/1812.07252/) (2018)
- [Action Robust Reinforcement Learning and Applications in Continuous Control](https://mlsanity.com/paper/1901.09184/) (2019)

## Built on by (most cited first, 30 of 38)

- [Uncertainty Quantification and Confidence Calibration in Large Language Models: A Survey](https://mlsanity.com/paper/2503.15850/) (2025)
- [WebSailor-V2: Bridging the Chasm to Proprietary Agents via Synthetic Data and Scalable Reinforcement Learning](https://mlsanity.com/paper/2509.13305/) (2025)
- [Residual Off-Policy RL for Finetuning Behavior Cloning Policies](https://mlsanity.com/paper/2509.19301/) (2025)
- [Scaling Agent Learning via Experience Synthesis](https://mlsanity.com/paper/2511.03773/) (2025)
- [SSRL: Self-Search Reinforcement Learning](https://mlsanity.com/paper/2508.10874/) (2025)
- [Generative AI in Transportation Planning: A Survey](https://mlsanity.com/paper/2503.07158/) (2025)
- [Data Pyramid for Embodied Manipulation: A Survey](https://mlsanity.com/paper/2607.24744/) (2026)
- [Linear Mixture Distributionally Robust Markov Decision Processes](https://mlsanity.com/paper/2505.18044/) (2025)
- [DeepShade: Enable Shade Simulation by Text-conditioned Image Generation](https://mlsanity.com/paper/2507.12103/) (2025)
- [Conformal Feedback Alignment: Quantifying Answer-Level Reliability for Robust LLM Alignment](https://mlsanity.com/paper/2601.17329/) (2026)
- [A Multi-Fidelity Control Variate Approach for Policy Gradient Estimation](https://mlsanity.com/paper/2503.05696/) (2025)
- [Diffusion-Based Forecasting for Uncertainty-Aware Model Predictive Control](https://mlsanity.com/paper/2503.15095/) (2025)
- [Beware Untrusted Simulators -- Reward-Free Backdoor Attacks in Reinforcement Learning](https://mlsanity.com/paper/2602.05089/) (2026)
- [Computational Pathology in the Era of Emerging Foundation and Agentic AI -- International Expert Perspectives on Clinical Integration and Translational Readiness](https://mlsanity.com/paper/2603.05884/) (2026)
- [Learning human-to-robot handovers through 3D scene reconstruction](https://mlsanity.com/paper/2507.08726/) (2025)
- [Bi-Level Reinforcement Learning Pathway for Sim-to-Real Optimality](https://mlsanity.com/paper/2510.17709/) (2025)
- [Sim-MSTNet: sim2real based Multi-task SpatioTemporal Network Traffic Forecasting](https://mlsanity.com/paper/2601.21384/) (2026)
- [Enhancing Goal Inference via Correction Timing](https://mlsanity.com/paper/2602.18603/) (2026)
- [The Sim-to-Real Gap of Foundation Model Agents: A Unified MDP Perspective](https://mlsanity.com/paper/2606.07017/) (2026)
- [Targeting World Models to Compromise Robot Learning Pipelines](https://mlsanity.com/paper/2606.09499/) (2026)
- [Tac-DINO: Learning Vision-Tactile Features with Patch Alignment](https://mlsanity.com/paper/2606.12069/) (2026)
- [AndroidReality: How Far Are Mobile Agents from the Real World?](https://mlsanity.com/paper/2608.07775/) (2026)
- [Joint-Local Grounded Action Transformation for Sim-to-Real Transfer in Multi-Agent Traffic Control](https://mlsanity.com/paper/2507.15174/) (2025)
- [Rationality Measurement and Theory for Reinforcement Learning Agents](https://mlsanity.com/paper/2602.04737/) (2026)
- [Online Adaptive Reinforcement Learning with Echo State Networks for Non-Stationary Dynamics](https://mlsanity.com/paper/2602.06326/) (2026)
- [Efficient Unsupervised Environment Design through Hierarchical Policy Representation Learning](https://mlsanity.com/paper/2602.09813/) (2026)
- [Developing AI Agents with Simulated Data: Why, what, and how?](https://mlsanity.com/paper/2602.15816/) (2026)
- [Automatic Generation of High-Performance RL Environments](https://mlsanity.com/paper/2603.12145/) (2026)
- [Mitigating Data Scarcity in Spaceflight Applications for Offline Reinforcement Learning Using Physics-Informed Deep Generative Models](https://mlsanity.com/paper/2604.02438/) (2026)
- [Application of Deep Reinforcement Learning to Event-Triggered Control for Networked Artificial Pancreas Systems](https://mlsanity.com/paper/2604.26126/) (2026)

## Abstract (arXiv)

Deep Reinforcement Learning (RL) has been explored and verified to be effective in solving decision-making tasks in various domains, such as robotics, transportation, recommender systems, etc. It learns from the interaction with environments and updates the policy using the collected experience. However, due to the limited real-world data and unbearable consequences of taking detrimental actions, the learning of RL policy is mainly restricted within the simulators. This practice guarantees safety in learning but introduces an inevitable sim-to-real gap in terms of deployment, thus causing degraded performance and risks in execution. There are attempts to solve the sim-to-real problems from different domains with various techniques, especially in the era with emerging techniques such as large foundations or language models that have cast light on the sim-to-real. This survey paper, to the best of our knowledge, is the first taxonomy that formally frames the sim-to-real techniques from key elements of the Markov Decision Process (State, Action, Transition, and Reward). Based on the framework, we cover comprehensive literature from the classic to the most advanced methods including the sim-to-real techniques empowered by foundation models, and we also discuss the specialties that are worth attention in different domains of sim-to-real problems. Then we summarize the formal evaluation process of sim-to-real performance with accessible code or benchmarks. The challenges and opportunities are also presented to encourage future exploration of this direction. We are actively maintaining a repository to include the most up-to-date sim-to-real research work to help domain researchers.