mlsanityData to 13 Oct 2026

Differentiable Physics Models for Real-world Offline Model-based Reinforcement Learning

arXiv 2011.01734 ↗First posted 3 Nov 2020cs.RO cs.LGcited by 27

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

A limitation of model-based reinforcement learning (MBRL) is the exploitation of errors in the learned models. Black-box models can fit complex dynamics with high fidelity, but their behavior is undefined outside of the data distribution.Physics-based models are better at extrapolating, due to the general validity of their informed structure, but underfit in the real world due to the presence of unmodeled phenomena. In this work, we demonstrate experimentally that for the offline model-based reinforcement learning setting, physics-based models can be beneficial compared to high-capacity function approximators if the mechanical structure is known. Physics-based models can learn to perform the ball in a cup (BiC) task on a physical manipulator using only 4 minutes of sampled data using offline MBRL. We find that black-box models consistently produce unviable policies for BiC as all predicted trajectories diverge to physically impossible state, despite having access to more data than the physics-based model. In addition, we generalize the approach of physics parameter identification from modeling holonomic multi-body systems to systems with nonholonomic dynamics using end-to-end automatic differentiation. Videos: https://sites.google.com/view/ball-in-a-cup-in-4-minutes/

Key facts. Cited by 27 papers in the map (data through 13 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 30 of its 52 references that are in the map. First posted on arXiv 3 Nov 2020.

Builds on

30 of 52 references.

Closest ancestors · 1

Same direction, newest first

  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

Other references · 29

  1. 2020Learning to Simulate Complex Physics with Graph Networkscited by 873
  2. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 756
  3. 2018Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Modelscited by 754
  4. 2019Hamiltonian Neural Networkscited by 705
  5. 2019Learning agile and dynamic motor skills for legged robotscited by 406
  6. 2020Lagrangian Neural Networkscited by 392
  7. 2019DiffTaichi: Differentiable Programming for Physical Simulationcited by 252
  8. 2018Graph networks as learnable physics engines for inference and controlcited by 243
  9. 2019Deep Lagrangian Networks: Using Physics as Model Prior for Deep Learningcited by 207
  10. 2018Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experiencecited by 203
  11. 2019Symplectic ODE-Net: Learning Hamiltonian Dynamics with Controlcited by 184
  12. 2019Benchmarking Model-Based Reinforcement Learningcited by 158
17 more
  1. 2016A Differentiable Physics Engine for Deep Learning in Roboticscited by 107
  2. 2019BayesSim: adaptive domain randomization via probabilistic inference for robotics simulatorscited by 79
  3. 2019Variational Integrator Networks for Physically Structured Embeddingscited by 36
  4. 2019Deep Lagrangian Networks for end-to-end learning of energy-based control for under-actuated systemscited by 32
  5. 2019Assessing Transferability from Simulation to Reality for Reinforcement Learningcited by 30
  6. 2019Self-Paced Contextual Reinforcement Learningcited by 26
  7. 2020Encoding Physical Constraints in Differentiable Newton-Euler Algorithmcited by 23
  8. 2019TuneNet: One-Shot Residual Tuning for System Identification and Sim-to-Real Robot Task Transfercited by 21
  9. 2019Interactive Differentiable Simulationcited by 17
  10. 2020ADD: Analytically Differentiable Dynamics for Multi-Body Systems with Frictional Contactcited by 17
  11. 2020High Acceleration Reinforcement Learning for Real-World Juggling with Binary Rewardscited by 14
  12. 2019A General Framework for Structured Learning of Mechanical Systemscited by 13
  13. 2017Linear Matrix Inequalities for Physically-Consistent Inertial Parameter Identification: A Statistical Perspective on the Mass Distributioncited by 11
  14. 2016Identification of Fully Physical Consistent Inertial Parameters using Optimization on Manifoldscited by 8
  15. 2019Simultaneously Learning Vision and Feature-based Control Policies for Real-world Ball-in-a-Cupcited by 6
  16. 2020A Differentiable Newton Euler Algorithm for Multi-body Model Learningcited by 6
  17. 2020Learning to Play Cup-and-Ball with Noisy Camera Observationscited by 1

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 3; 2022: 7; 2023: 5; 2024: 4; 2025: 6; 2026: 2.

27 papers.

Sort:

By area:

  1. 2022Physics-Informed Machine Learning: A Survey on Problems, Methods and Applicationscited by 90
  2. 2021Robot Learning from Randomized Simulations: A Reviewcited by 56
  3. 2022DiffCloud: Real-to-Sim from Point Clouds with Differentiable Simulation and Rendering of Deformable Objectscited by 29
  4. 2023A Survey on Physics Informed Reinforcement Learning: Review and Open Problemscited by 22
  5. 2021Learning Dynamics Models for Model Predictive Agentscited by 21
  6. 2022Rethinking Optimization with Differentiable Simulation from a Global Perspectivecited by 21
  7. 2021Combining Physics and Deep Learning to learn Continuous-Time Dynamics Modelscited by 20
  8. 2023TWIST: Teacher-Student World Model Distillation for Efficient Sim-to-Real Transfercited by 19
  9. 2022Inferring Smooth Control: Monte Carlo Posterior Policy Iteration with Gaussian Processescited by 10
  10. 2025Accelerating Model-Based Reinforcement Learning with State-Space World Modelscited by 10
  11. 2022j-Wave: An open-source differentiable wave simulatorcited by 9
  12. 2022Learning Tool Morphology for Contact-Rich Manipulation Tasks with Differentiable Simulationcited by 9
15 more
  1. 2023Diminishing Return of Value Expansion Methods in Model-Based Reinforcement Learningcited by 5
  2. 2024Learning Object Properties Using Robot Proprioception via Differentiable Robot-Object Interactioncited by 3
  3. 2022A Recurrent Differentiable Engine for Modeling Tensegrity Robots Trainable with Low-Frequency Datacited by 2
  4. 2024Enhanced Prediction of Multi-Agent Trajectories via Control Inference and State-Space Dynamicscited by 2
  5. 2024Discovering Artificial Viscosity Models for Discontinuous Galerkin Approximation of Conservation Laws using Physics-Informed Machine Learningcited by 1
  6. 2025Unlocking Efficient Vehicle Dynamics Modeling via Analytic World Modelscited by 1
  7. 2025Newtonian and Lagrangian Neural Networks: A Comparison Towards Efficient Inverse Dynamics Identificationcited by 1
  8. 2025Autonomous Vehicle Path Planning by Searching With Differentiable Simulationcited by 1
  9. 2026DreamSAC: Learning Hamiltonian World Models via Symmetry Explorationcited by 1
  10. 2023Data-efficient, Explainable and Safe Box Manipulation: Illustrating the Advantages of Physical Priors in Model-Predictive Controlcited by 0
  11. 2023On Physical Origins of Learningcited by 0
  12. 2024Diminishing Return of Value Expansion Methodscited by 0
  13. 2025Reinforcement Twinning for Hybrid Control of Flapping-Wing Dronescited by 0
  14. 2025Diff-MSM: Differentiable MusculoSkeletal Model for Simultaneous Identification of Human Muscle and Bone Parameterscited by 0
  15. 2026VertAX: a differentiable vertex model for learning epithelial tissue mechanicscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021A Differentiable Newton-Euler Algorithm for Real-World Robotics
  2. 2025Uncertainty-Aware Robotic World Model Makes Offline Model-Based Reinforcement Learning Work on Real Robots
  3. 2025PIN-WM: Learning Physics-INformed World Models for Non-Prehensile Manipulation
  4. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learning
  5. 2023A Survey on Offline Model-Based Reinforcement Learning
  6. 2024Offline Model-Based Reinforcement Learning with Anti-Exploration
  7. 2025Integrating Learning-Based Manipulation and Physics-Based Locomotion for Whole-Body Badminton Robot Control
  8. 2026Cross-Embodiment Offline Reinforcement Learning for Heterogeneous Robot Datasets