# Assessing Transferability from Simulation to Reality for Reinforcement Learning

arXiv: https://arxiv.org/abs/1907.04685 · HTML: https://mlsanity.com/paper/1907.04685/

## Key facts

- Cited by 33 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Jul 2019.

## Built on by (most cited first, 30 of 33)

- [Robust Reinforcement Learning on State Observations with Learned Optimal Adversary](https://mlsanity.com/paper/2101.08452/) (2021)
- [Robot Learning from Randomized Simulations: A Review](https://mlsanity.com/paper/2111.00956/) (2021)
- [A Survey of Embodied Learning for Object-Centric Robotic Manipulation](https://mlsanity.com/paper/2408.11537/) (2024)
- [ASID: Active Exploration for System Identification in Robotic Manipulation](https://mlsanity.com/paper/2404.12308/) (2024)
- [Differentiable Physics Models for Real-world Offline Model-based Reinforcement Learning](https://mlsanity.com/paper/2011.01734/) (2020)
- [The Reality Gap in Robotics: Challenges, Solutions, and Best Practices](https://mlsanity.com/paper/2510.20808/) (2025)
- [DROPO: Sim-to-Real Transfer with Offline Domain Randomization](https://mlsanity.com/paper/2201.08434/) (2022)
- [Domain Randomization via Entropy Maximization](https://mlsanity.com/paper/2311.01885/) (2023)
- [A Dirichlet Process Mixture of Robust Task Models for Scalable Lifelong Reinforcement Learning](https://mlsanity.com/paper/2205.10787/) (2022)
- [Reinforcement Learning from Simulation to Real World Autonomous Driving using Digital Twin](https://mlsanity.com/paper/2211.14874/) (2022)
- [Procedural Content Generation: Better Benchmarks for Transfer Reinforcement Learning](https://mlsanity.com/paper/2105.14780/) (2021)
- [Improve Robustness of Reinforcement Learning against Observation Perturbations via l_∞ Lipschitz Policy Networks](https://mlsanity.com/paper/2312.08751/) (2023)
- [Digital Twin Assisted Deep Reinforcement Learning for Online Admission Control in Sliced Network](https://mlsanity.com/paper/2310.09299/) (2023)
- [Online vs. Offline Adaptive Domain Randomization Benchmark](https://mlsanity.com/paper/2206.14661/) (2022)
- [DynamicPAE: Generating Scene-Aware Physical Adversarial Examples in Real-Time](https://mlsanity.com/paper/2412.08053/) (2024)
- [Flow-based Domain Randomization for Learning and Sequencing Robotic Skills](https://mlsanity.com/paper/2502.01800/) (2025)
- [On the Emergence of Whole-body Strategies from Humanoid Robot Push-recovery Learning](https://mlsanity.com/paper/2104.14534/) (2021)
- [Distilled Domain Randomization](https://mlsanity.com/paper/2112.03149/) (2021)
- [A Conservative Approach for Few-Shot Transfer in Off-Dynamics Reinforcement Learning](https://mlsanity.com/paper/2312.15474/) (2023)
- [Bridging the Reality Gap of Reinforcement Learning based Traffic Signal Control using Domain Randomization and Meta Learning](https://mlsanity.com/paper/2307.11357/) (2023)
- [Learning Based NMPC Adaptation for Autonomous Driving using Parallelized Digital Twin](https://mlsanity.com/paper/2402.16645/) (2024)
- [Self-organized free-flight arrival for urban air mobility](https://mlsanity.com/paper/2404.03710/) (2024)
- [Domain Curiosity: Learning Efficient Data Collection Strategies for Domain Adaptation](https://mlsanity.com/paper/2103.07223/) (2021)
- [Velocity-History-Based Soft Actor-Critic Tackling IROS'24 Competition "AI Olympics with RealAIGym"](https://mlsanity.com/paper/2410.20096/) (2024)
- [Enhancing Deep Reinforcement Learning-based Robot Navigation Generalization through Scenario Augmentation](https://mlsanity.com/paper/2503.01146/) (2025)
- [Post-Convergence Sim-to-Real Policy Transfer: A Principled Alternative to Cherry-Picking](https://mlsanity.com/paper/2504.15414/) (2025)
- [Action Robust Reinforcement Learning via Optimal Adversary Aware Policy Optimization](https://mlsanity.com/paper/2507.03372/) (2025)
- [Configuration Path Control](https://mlsanity.com/paper/2204.02471/) (2022)
- [An Empirical Analysis of the Use of Real-Time Reachability for the Safety Assurance of Autonomous Vehicles](https://mlsanity.com/paper/2205.01419/) (2022)
- [Zero-shot Transfer of Reinforcement Learning Control Policies for the Swing-Up and Stabilization of a Cart-Pole System](https://mlsanity.com/paper/2606.22145/) (2026)

## Abstract (arXiv)

Learning robot control policies from physics simulations is of great interest to the robotics community as it may render the learning process faster, cheaper, and safer by alleviating the need for expensive real-world experiments. However, the direct transfer of learned behavior from simulation to reality is a major challenge. Optimizing a policy on a slightly faulty simulator can easily lead to the maximization of the `Simulation Optimization Bias` (SOB). In this case, the optimizer exploits modeling errors of the simulator such that the resulting behavior can potentially damage the robot. We tackle this challenge by applying domain randomization, i.e., randomizing the parameters of the physics simulations during learning. We propose an algorithm called Simulation-based Policy Optimization with Transferability Assessment (SPOTA) which uses an estimator of the SOB to formulate a stopping criterion for training. The introduced estimator quantifies the over-fitting to the set of domains experienced while training. Our experimental results on two different second order nonlinear systems show that the new simulation-based policy search algorithm is able to learn a control policy exclusively from a randomized simulator, which can be applied directly to real systems without any additional training.