# Differentiable Physics Models for Real-world Offline Model-based Reinforcement Learning

arXiv: https://arxiv.org/abs/2011.01734 · HTML: https://mlsanity.com/paper/2011.01734/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 30 of its 52 references that are in the map.
- First posted on arXiv 3 Nov 2020.

## Placed in

- Offline RL and imitation learning from data / [Offline RL algorithms](https://mlsanity.com/offlinerl/offline-rl-algorithms/)

## Builds on (most cited first, 30 of 30)

- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Learning to Simulate Complex Physics with Graph Networks](https://mlsanity.com/paper/2002.09405/) (2020)
- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [When to Trust Your Model: Model-Based Policy Optimization](https://mlsanity.com/paper/1906.08253/) (2019)
- [Hamiltonian Neural Networks](https://mlsanity.com/paper/1906.01563/) (2019)
- [Learning agile and dynamic motor skills for legged robots](https://mlsanity.com/paper/1901.08652/) (2019)
- [Lagrangian Neural Networks](https://mlsanity.com/paper/2003.04630/) (2020)
- [DiffTaichi: Differentiable Programming for Physical Simulation](https://mlsanity.com/paper/1910.00935/) (2019)
- [Graph networks as learnable physics engines for inference and control](https://mlsanity.com/paper/1806.01242/) (2018)
- [Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience](https://mlsanity.com/paper/1810.05687/) (2018)
- [Deep Lagrangian Networks: Using Physics as Model Prior for Deep Learning](https://mlsanity.com/paper/1907.04490/) (2019)
- [Symplectic ODE-Net: Learning Hamiltonian Dynamics with Control](https://mlsanity.com/paper/1909.12077/) (2019)
- [Benchmarking Model-Based Reinforcement Learning](https://mlsanity.com/paper/1907.02057/) (2019)
- [A Differentiable Physics Engine for Deep Learning in Robotics](https://mlsanity.com/paper/1611.01652/) (2016)
- [BayesSim: adaptive domain randomization via probabilistic inference for robotics simulators](https://mlsanity.com/paper/1906.01728/) (2019)
- [Variational Integrator Networks for Physically Structured Embeddings](https://mlsanity.com/paper/1910.09349/) (2019)
- [Deep Lagrangian Networks for end-to-end learning of energy-based control for under-actuated systems](https://mlsanity.com/paper/1907.04489/) (2019)
- [Assessing Transferability from Simulation to Reality for Reinforcement Learning](https://mlsanity.com/paper/1907.04685/) (2019)
- [Self-Paced Contextual Reinforcement Learning](https://mlsanity.com/paper/1910.02826/) (2019)
- [Encoding Physical Constraints in Differentiable Newton-Euler Algorithm](https://mlsanity.com/paper/2001.08861/) (2020)
- [TuneNet: One-Shot Residual Tuning for System Identification and Sim-to-Real Robot Task Transfer](https://mlsanity.com/paper/1907.11200/) (2019)
- [Interactive Differentiable Simulation](https://mlsanity.com/paper/1905.10706/) (2019)
- [ADD: Analytically Differentiable Dynamics for Multi-Body Systems with Frictional Contact](https://mlsanity.com/paper/2007.00987/) (2020)
- [High Acceleration Reinforcement Learning for Real-World Juggling with Binary Rewards](https://mlsanity.com/paper/2010.13483/) (2020)
- [A General Framework for Structured Learning of Mechanical Systems](https://mlsanity.com/paper/1902.08705/) (2019)
- [Linear Matrix Inequalities for Physically-Consistent Inertial Parameter Identification: A Statistical Perspective on the Mass Distribution](https://mlsanity.com/paper/1701.04395/) (2017)
- [Identification of Fully Physical Consistent Inertial Parameters using Optimization on Manifolds](https://mlsanity.com/paper/1610.08703/) (2016)
- [Simultaneously Learning Vision and Feature-based Control Policies for Real-world Ball-in-a-Cup](https://mlsanity.com/paper/1902.04706/) (2019)
- [A Differentiable Newton Euler Algorithm for Multi-body Model Learning](https://mlsanity.com/paper/2010.09802/) (2020)
- [Learning to Play Cup-and-Ball with Noisy Camera Observations](https://mlsanity.com/paper/2007.09562/) (2020)

## Built on by (most cited first, 27 of 27)

- [Physics-Informed Machine Learning: A Survey on Problems, Methods and Applications](https://mlsanity.com/paper/2211.08064/) (2022)
- [Robot Learning from Randomized Simulations: A Review](https://mlsanity.com/paper/2111.00956/) (2021)
- [DiffCloud: Real-to-Sim from Point Clouds with Differentiable Simulation and Rendering of Deformable Objects](https://mlsanity.com/paper/2204.03139/) (2022)
- [A Survey on Physics Informed Reinforcement Learning: Review and Open Problems](https://mlsanity.com/paper/2309.01909/) (2023)
- [Rethinking Optimization with Differentiable Simulation from a Global Perspective](https://mlsanity.com/paper/2207.00167/) (2022)
- [Learning Dynamics Models for Model Predictive Agents](https://mlsanity.com/paper/2109.14311/) (2021)
- [Combining Physics and Deep Learning to learn Continuous-Time Dynamics Models](https://mlsanity.com/paper/2110.01894/) (2021)
- [TWIST: Teacher-Student World Model Distillation for Efficient Sim-to-Real Transfer](https://mlsanity.com/paper/2311.03622/) (2023)
- [Inferring Smooth Control: Monte Carlo Posterior Policy Iteration with Gaussian Processes](https://mlsanity.com/paper/2210.03512/) (2022)
- [Accelerating Model-Based Reinforcement Learning with State-Space World Models](https://mlsanity.com/paper/2502.20168/) (2025)
- [j-Wave: An open-source differentiable wave simulator](https://mlsanity.com/paper/2207.01499/) (2022)
- [Learning Tool Morphology for Contact-Rich Manipulation Tasks with Differentiable Simulation](https://mlsanity.com/paper/2211.02201/) (2022)
- [Diminishing Return of Value Expansion Methods in Model-Based Reinforcement Learning](https://mlsanity.com/paper/2303.03955/) (2023)
- [Learning Object Properties Using Robot Proprioception via Differentiable Robot-Object Interaction](https://mlsanity.com/paper/2410.03920/) (2024)
- [A Recurrent Differentiable Engine for Modeling Tensegrity Robots Trainable with Low-Frequency Data](https://mlsanity.com/paper/2203.00041/) (2022)
- [Enhanced Prediction of Multi-Agent Trajectories via Control Inference and State-Space Dynamics](https://mlsanity.com/paper/2408.12609/) (2024)
- [Discovering Artificial Viscosity Models for Discontinuous Galerkin Approximation of Conservation Laws using Physics-Informed Machine Learning](https://mlsanity.com/paper/2402.16517/) (2024)
- [Unlocking Efficient Vehicle Dynamics Modeling via Analytic World Models](https://mlsanity.com/paper/2502.10012/) (2025)
- [Newtonian and Lagrangian Neural Networks: A Comparison Towards Efficient Inverse Dynamics Identification](https://mlsanity.com/paper/2506.17994/) (2025)
- [Autonomous Vehicle Path Planning by Searching With Differentiable Simulation](https://mlsanity.com/paper/2511.11043/) (2025)
- [DreamSAC: Learning Hamiltonian World Models via Symmetry Exploration](https://mlsanity.com/paper/2603.07545/) (2026)
- [Data-efficient, Explainable and Safe Box Manipulation: Illustrating the Advantages of Physical Priors in Model-Predictive Control](https://mlsanity.com/paper/2303.01563/) (2023)
- [On Physical Origins of Learning](https://mlsanity.com/paper/2310.02375/) (2023)
- [Diminishing Return of Value Expansion Methods](https://mlsanity.com/paper/2412.20537/) (2024)
- [Reinforcement Twinning for Hybrid Control of Flapping-Wing Drones](https://mlsanity.com/paper/2505.18201/) (2025)
- [Diff-MSM: Differentiable MusculoSkeletal Model for Simultaneous Identification of Human Muscle and Bone Parameters](https://mlsanity.com/paper/2508.13303/) (2025)
- [VertAX: a differentiable vertex model for learning epithelial tissue mechanics](https://mlsanity.com/paper/2604.06896/) (2026)

## Abstract (arXiv)

A limitation of model-based reinforcement learning (MBRL) is the exploitation of errors in the learned models. Black-box models can fit complex dynamics with high fidelity, but their behavior is undefined outside of the data distribution.Physics-based models are better at extrapolating, due to the general validity of their informed structure, but underfit in the real world due to the presence of unmodeled phenomena. In this work, we demonstrate experimentally that for the offline model-based reinforcement learning setting, physics-based models can be beneficial compared to high-capacity function approximators if the mechanical structure is known. Physics-based models can learn to perform the ball in a cup (BiC) task on a physical manipulator using only 4 minutes of sampled data using offline MBRL. We find that black-box models consistently produce unviable policies for BiC as all predicted trajectories diverge to physically impossible state, despite having access to more data than the physics-based model. In addition, we generalize the approach of physics parameter identification from modeling holonomic multi-body systems to systems with nonholonomic dynamics using end-to-end automatic differentiation. Videos: https://sites.google.com/view/ball-in-a-cup-in-4-minutes/