# DROPO: Sim-to-Real Transfer with Offline Domain Randomization

arXiv: https://arxiv.org/abs/2201.08434 · HTML: https://mlsanity.com/paper/2201.08434/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 43 references that are in the map.
- First posted on arXiv 20 Jan 2022.

## Builds on (most cited first, 30 of 30)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [End-to-End Training of Deep Visuomotor Policies](https://mlsanity.com/paper/1504.00702/) (2015)
- [Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction](https://mlsanity.com/paper/1906.00949/) (2019)
- [Learning Dexterous In-Hand Manipulation](https://mlsanity.com/paper/1808.00177/) (2018)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey](https://mlsanity.com/paper/2009.13303/) (2020)
- [Sim-to-Real: Learning Agile Locomotion For Quadruped Robots](https://mlsanity.com/paper/1804.10332/) (2018)
- [Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learning](https://mlsanity.com/paper/1803.11347/) (2018)
- [CAD2RL: Real Single-Image Flight without a Single Real Image](https://mlsanity.com/paper/1611.04201/) (2016)
- [Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience](https://mlsanity.com/paper/1810.05687/) (2018)
- [Active Domain Randomization](https://mlsanity.com/paper/1904.04762/) (2019)
- [EPOpt: Learning Robust Neural Network Policies Using Model Ensembles](https://mlsanity.com/paper/1610.01283/) (2016)
- [BayesSim: adaptive domain randomization via probabilistic inference for robotics simulators](https://mlsanity.com/paper/1906.01728/) (2019)
- [Meta Reinforcement Learning for Sim-to-real Domain Adaptation](https://mlsanity.com/paper/1909.12906/) (2019)
- [Auto-Tuned Sim-to-Real Transfer](https://mlsanity.com/paper/2104.07662/) (2021)
- [Data-efficient Domain Randomization with Bayesian Optimization](https://mlsanity.com/paper/2003.02471/) (2020)
- [Assessing Transferability from Simulation to Reality for Reinforcement Learning](https://mlsanity.com/paper/1907.04685/) (2019)
- [Deep Predictive Policy Training using Reinforcement Learning](https://mlsanity.com/paper/1703.00727/) (2017)
- [Crossing The Gap: A Deep Dive into Zero-Shot Sim-to-Real Transfer for Dynamics](https://mlsanity.com/paper/2008.06686/) (2020)
- [Sim2Real View Invariant Visual Servoing by Recurrent Control](https://mlsanity.com/paper/1712.07642/) (2017)
- [Affordance Learning for End-to-End Visuomotor Robot Control](https://mlsanity.com/paper/1903.04053/) (2019)
- [Reinforcement Learning for Pivoting Task](https://mlsanity.com/paper/1703.00472/) (2017)
- [DROID: Minimizing the Reality Gap using Single-Shot Human Demonstration](https://mlsanity.com/paper/2102.11003/) (2021)
- [A User's Guide to Calibrating Robotics Simulators](https://mlsanity.com/paper/2011.08985/) (2020)
- [Few-shot model-based adaptation in noisy conditions](https://mlsanity.com/paper/2010.08397/) (2020)
- [Domain Curiosity: Learning Efficient Data Collection Strategies for Domain Adaptation](https://mlsanity.com/paper/2103.07223/) (2021)

## Built on by (most cited first, 19 of 19)

- [The Reality Gap in Robotics: Challenges, Solutions, and Best Practices](https://mlsanity.com/paper/2510.20808/) (2025)
- [Domain Randomization via Entropy Maximization](https://mlsanity.com/paper/2311.01885/) (2023)
- [Continual Domain Randomization](https://mlsanity.com/paper/2403.12193/) (2024)
- [Constrained Reinforcement Learning using Distributional Representation for Trustworthy Quadrotor UAV Tracking Control](https://mlsanity.com/paper/2302.11694/) (2023)
- [TAR: Teacher-Aligned Representations via Contrastive Learning for Quadrupedal Locomotion](https://mlsanity.com/paper/2503.20839/) (2025)
- [Online vs. Offline Adaptive Domain Randomization Benchmark](https://mlsanity.com/paper/2206.14661/) (2022)
- [Self-evolved Imitation Learning in Simulated World](https://mlsanity.com/paper/2509.19460/) (2025)
- [Domain Randomization for Robust, Affordable and Effective Closed-loop Control of Soft Robots](https://mlsanity.com/paper/2303.04136/) (2023)
- [Safe Continual Domain Adaptation after Sim2Real Transfer of Reinforcement Learning Policies in Robotics](https://mlsanity.com/paper/2503.10949/) (2025)
- [Shaping Laser Pulses with Reinforcement Learning](https://mlsanity.com/paper/2503.00499/) (2025)
- [Belief-Based Offline Reinforcement Learning for Delay-Robust Policy Optimization](https://mlsanity.com/paper/2506.00131/) (2025)
- [Statistical Guarantees for Offline Domain Randomization](https://mlsanity.com/paper/2506.10133/) (2025)
- [NavRL++: A System-Level Framework for Improving Sim-to-Real Transfer in Reinforcement Learning-Based Robot Navigation](https://mlsanity.com/paper/2605.15559/) (2026)
- [DiSA-IQL: Offline Reinforcement Learning for Robust Soft Robot Control under Distribution Shifts](https://mlsanity.com/paper/2510.00358/) (2025)
- [Grounding Sim-to-Real Generalization in Robotic Manipulation: An Empirical Study with Vision-Language-Action Models](https://mlsanity.com/paper/2603.22876/) (2026)
- [Wiggle and Go! System Identification for Zero-Shot Dynamic Rope Manipulation](https://mlsanity.com/paper/2604.22102/) (2026)
- [A Unified Causal-Origin Taxonomy of Distributional Shifts in Reinforcement Learning](https://mlsanity.com/paper/2606.16933/) (2026)
- [How Should a Simulation-to-Reality Transfer Budget Be Spent?](https://mlsanity.com/paper/2606.22062/) (2026)
- [Enhancing Sim2Real Transfer for Torque-Controlled Robots through Real2Sim Dynamics Estimation and Reinforcement Learning](https://mlsanity.com/paper/2608.22629/) (2026)

## Abstract (arXiv)

In recent years, domain randomization over dynamics parameters has gained a lot of traction as a method for sim-to-real transfer of reinforcement learning policies in robotic manipulation; however, finding optimal randomization distributions can be difficult. In this paper, we introduce DROPO, a novel method for estimating domain randomization distributions for safe sim-to-real transfer. Unlike prior work, DROPO only requires a limited, precollected offline dataset of trajectories, and explicitly models parameter uncertainty to match real data using a likelihood-based approach. We demonstrate that DROPO is capable of recovering dynamic parameter distributions in simulation and finding a distribution capable of compensating for an unmodeled phenomenon. We also evaluate the method in two zero-shot sim-to-real transfer scenarios, showing successful domain transfer and improved performance over prior methods.