# Imitation Learning via Differentiable Physics

arXiv: https://arxiv.org/abs/2206.04873 · HTML: https://mlsanity.com/paper/2206.04873/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 20 of its 33 references that are in the map.
- First posted on arXiv 10 Jun 2022.

## Placed in

- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Builds on (most cited first, 20 of 20)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020)
- [MOPO: Model-based Offline Policy Optimization](https://mlsanity.com/paper/2005.13239/) (2020)
- [Learning Robust Rewards with Adversarial Inverse Reinforcement Learning](https://mlsanity.com/paper/1710.11248/) (2017)
- [Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation](https://mlsanity.com/paper/2106.13281/) (2021)
- [Transporter Networks: Rearranging the Visual World for Robotic Manipulation](https://mlsanity.com/paper/2010.14406/) (2020)
- [Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learning](https://mlsanity.com/paper/1809.02925/) (2018)
- [Primal Wasserstein Imitation Learning](https://mlsanity.com/paper/2006.04678/) (2020)
- [PlasticineLab: A Soft-Body Manipulation Benchmark with Differentiable Physics](https://mlsanity.com/paper/2104.03311/) (2021)
- [Scalable Differentiable Physics for Learning and Control](https://mlsanity.com/paper/2007.02168/) (2020)
- [Off-Policy Imitation Learning from Observations](https://mlsanity.com/paper/2102.13185/) (2021)
- [DiSECt: A Differentiable Simulation Engine for Autonomous Robotic Cutting](https://mlsanity.com/paper/2105.12244/) (2021)
- [Model-Augmented Actor-Critic: Backpropagating through Paths](https://mlsanity.com/paper/2005.08068/) (2020)
- [DiffSkill: Skill Abstraction from Differentiable Physics for Deformable Object Manipulations with Tools](https://mlsanity.com/paper/2203.17275/) (2022)
- [Contrastive Variational Reinforcement Learning for Complex Observations](https://mlsanity.com/paper/2008.02430/) (2020)
- [OPIRL: Sample Efficient Off-Policy Inverse Reinforcement Learning via Distribution Matching](https://mlsanity.com/paper/2109.04307/) (2021)
- [Ab Initio Particle-based Object Manipulation](https://mlsanity.com/paper/2107.08865/) (2021)

## Built on by (most cited first, 2 of 2)

- [DiffMimic: Efficient Motion Mimicking with Differentiable Physics](https://mlsanity.com/paper/2304.03274/) (2023)
- [Human Motion Prediction under Unexpected Perturbation](https://mlsanity.com/paper/2403.15891/) (2024)

## Abstract (arXiv)

Existing imitation learning (IL) methods such as inverse reinforcement learning (IRL) usually have a double-loop training process, alternating between learning a reward function and a policy and tend to suffer long training time and high variance. In this work, we identify the benefits of differentiable physics simulators and propose a new IL method, i.e., Imitation Learning via Differentiable Physics (ILD), which gets rid of the double-loop design and achieves significant improvements in final performance, convergence speed, and stability. The proposed ILD incorporates the differentiable physics simulator as a physics prior into its computational graph for policy learning. It unrolls the dynamics by sampling actions from a parameterized policy, simply minimizing the distance between the expert trajectory and the agent trajectory, and back-propagating the gradient into the policy via temporal physics operators. With the physics prior, ILD policies can not only be transferable to unseen environment specifications but also yield higher final performance on a variety of tasks. In addition, ILD naturally forms a single-loop structure, which significantly improves the stability and training speed. To simplify the complex optimization landscape induced by temporal physics operations, ILD dynamically selects the learning objectives for each state during optimization. In our experiments, we show that ILD outperforms state-of-the-art methods in a variety of continuous control tasks with Brax, requiring only one expert demonstration. In addition, ILD can be applied to challenging deformable object manipulation tasks and can be generalized to unseen configurations.