# Transfer from Simulation to Real World through Learning Deep Inverse Dynamics Model

arXiv: https://arxiv.org/abs/1610.03518 · HTML: https://mlsanity.com/paper/1610.03518/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Oct 2016.

## Built on by (most cited first, 30 of 54)

- [State-Only Imitation Learning for Dexterous Manipulation](https://mlsanity.com/paper/2004.04650/) (2020)
- [Understanding Domain Randomization for Sim-to-real Transfer](https://mlsanity.com/paper/2110.03239/) (2021)
- [Robot Learning from Randomized Simulations: A Review](https://mlsanity.com/paper/2111.00956/) (2021)
- [A Policy-Guided Imitation Approach for Offline Reinforcement Learning](https://mlsanity.com/paper/2210.08323/) (2022)
- [Masked Autoencoding for Scalable and Generalizable Decision Making](https://mlsanity.com/paper/2211.12740/) (2022)
- [Learning Markov State Abstractions for Deep Reinforcement Learning](https://mlsanity.com/paper/2106.04379/) (2021)
- [CROP: Certifying Robust Policies for Reinforcement Learning through Functional Smoothing](https://mlsanity.com/paper/2106.09292/) (2021)
- [On Transforming Reinforcement Learning by Transformer: The Development Trajectory](https://mlsanity.com/paper/2212.14164/) (2022)
- [Polybot: Training One Policy Across Robots While Embracing Variability](https://mlsanity.com/paper/2307.03719/) (2023)
- [UniMASK: Unified Inference in Sequential Decision Problems](https://mlsanity.com/paper/2211.10869/) (2022)
- [The Reality Gap in Robotics: Challenges, Solutions, and Best Practices](https://mlsanity.com/paper/2510.20808/) (2025)
- [State Regularized Policy Optimization on Data with Dynamics Shift](https://mlsanity.com/paper/2306.03552/) (2023)
- [Pretraining in Deep Reinforcement Learning: A Survey](https://mlsanity.com/paper/2211.03959/) (2022)
- [Distributionally Robust Model-based Reinforcement Learning with Large State Spaces](https://mlsanity.com/paper/2309.02236/) (2023)
- [A Comprehensive Survey of Cross-Domain Policy Transfer for Embodied Agents](https://mlsanity.com/paper/2402.04580/) (2024)
- [Vision-based Learning for Drones: A Survey](https://mlsanity.com/paper/2312.05019/) (2023)
- [Provable Sim-to-real Transfer in Continuous Domain with Partial Observations](https://mlsanity.com/paper/2210.15598/) (2022)
- [On the Importance of Feature Decorrelation for Unsupervised Representation Learning in Reinforcement Learning](https://mlsanity.com/paper/2306.05637/) (2023)
- [DiAReL: Reinforcement Learning with Disturbance Awareness for Robust Sim2Real Policy Transfer in Robot Control](https://mlsanity.com/paper/2306.09010/) (2023)
- [Reward Certification for Policy Smoothed Reinforcement Learning](https://mlsanity.com/paper/2312.06436/) (2023)
- [Towards Flexible Inference in Sequential Decision Problems via Bidirectional Transformers](https://mlsanity.com/paper/2204.13326/) (2022)
- [Residual Model Learning for Microrobot Control](https://mlsanity.com/paper/2104.00631/) (2021)
- [Cloud-Edge Training Architecture for Sim-to-Real Deep Reinforcement Learning](https://mlsanity.com/paper/2203.02230/) (2022)
- [Learning What To Do by Simulating the Past](https://mlsanity.com/paper/2104.03946/) (2021)
- [rSoccer: A Framework for Studying Reinforcement Learning in Small and Very Small Size Robot Soccer](https://mlsanity.com/paper/2106.12895/) (2021)
- [Adversarial Diffusion for Robust Reinforcement Learning](https://mlsanity.com/paper/2509.23846/) (2025)
- [On the Emergence of Whole-body Strategies from Humanoid Robot Push-recovery Learning](https://mlsanity.com/paper/2104.14534/) (2021)
- [Dependability Analysis of Deep Reinforcement Learning based Robotics and Autonomous Systems through Probabilistic Model Checking](https://mlsanity.com/paper/2109.06523/) (2021)
- [Imitating, Fast and Slow: Robust learning from demonstrations via decision-time planning](https://mlsanity.com/paper/2204.03597/) (2022)
- [Robust Model-Based Reinforcement Learning with an Adversarial Auxiliary Model](https://mlsanity.com/paper/2406.09976/) (2024)

## Abstract (arXiv)

Developing control policies in simulation is often more practical and safer than directly running experiments in the real world. This applies to policies obtained from planning and optimization, and even more so to policies obtained from reinforcement learning, which is often very data demanding. However, a policy that succeeds in simulation often doesn't work when deployed on a real robot. Nevertheless, often the overall gist of what the policy does in simulation remains valid in the real world. In this paper we investigate such settings, where the sequence of states traversed in simulation remains reasonable for the real world, even if the details of the controls are not, as could be the case when the key differences lie in detailed friction, contact, mass and geometry properties. During execution, at each time step our approach computes what the simulation-based control policy would do, but then, rather than executing these controls on the real robot, our approach computes what the simulation expects the resulting next state(s) will be, and then relies on a learned deep inverse dynamics model to decide which real-world action is most suitable to achieve those next states. Deep models are only as good as their training data, and we also propose an approach for data collection to (incrementally) learn the deep inverse dynamics model. Our experiments shows our approach compares favorably with various baselines that have been developed for dealing with simulation to real world model discrepancy, including output error control and Gaussian dynamics adaptation.