# Physics-Informed Model-Based Reinforcement Learning

arXiv: https://arxiv.org/abs/2212.02179 · HTML: https://mlsanity.com/paper/2212.02179/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 28 references that are in the map.
- First posted on arXiv 5 Dec 2022.

## Builds on (most cited first, 21 of 21)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020)
- [When to Trust Your Model: Model-Based Policy Optimization](https://mlsanity.com/paper/1906.08253/) (2019)
- [Hamiltonian Neural Networks](https://mlsanity.com/paper/1906.01563/) (2019)
- [Differentiable Convex Optimization Layers](https://mlsanity.com/paper/1910.12430/) (2019)
- [Lagrangian Neural Networks](https://mlsanity.com/paper/2003.04630/) (2020)
- [Deep Lagrangian Networks: Using Physics as Model Prior for Deep Learning](https://mlsanity.com/paper/1907.04490/) (2019)
- [Symplectic ODE-Net: Learning Hamiltonian Dynamics with Control](https://mlsanity.com/paper/1909.12077/) (2019)
- [Learning Continuous Control Policies by Stochastic Value Gradients](https://mlsanity.com/paper/1510.09142/) (2015)
- [Simplifying Hamiltonian and Lagrangian Neural Networks via Explicit Constraints](https://mlsanity.com/paper/2010.13581/) (2020)
- [Dissipative SymODEN: Encoding Hamiltonian Dynamics with Dissipation and Control into Deep Learning](https://mlsanity.com/paper/2002.08860/) (2020)
- [Model-Augmented Actor-Critic: Backpropagating through Paths](https://mlsanity.com/paper/2005.08068/) (2020)
- [Benchmarking Energy-Conserving Neural Networks for Learning Dynamics from Data](https://mlsanity.com/paper/2012.02334/) (2020)
- [Extending Lagrangian and Hamiltonian Neural Networks with Differentiable Contact Models](https://mlsanity.com/paper/2102.06794/) (2021)
- [Deep Lagrangian Networks for end-to-end learning of energy-based control for under-actuated systems](https://mlsanity.com/paper/1907.04489/) (2019)

## Built on by (most cited first, 18 of 18)

- [From PINNs to PIKANs: Recent Advances in Physics-Informed Machine Learning](https://mlsanity.com/paper/2410.13228/) (2024)
- [A Survey on Physics Informed Reinforcement Learning: Review and Open Problems](https://mlsanity.com/paper/2309.01909/) (2023)
- [Deep Learning in Deterministic Computational Mechanics](https://mlsanity.com/paper/2309.15421/) (2023)
- [Physics-Informed Policy Optimization via Analytic Dynamics Regularization](https://mlsanity.com/paper/2603.14469/) (2026)
- [PH-Dreamer: A Physics-Driven World Model via Port-Hamiltonian Generative Dynamics](https://mlsanity.com/paper/2605.18303/) (2026)
- [Physics-Informed Model and Hybrid Planning for Efficient Dyna-Style Reinforcement Learning](https://mlsanity.com/paper/2407.02217/) (2024)
- [Sample-Efficient Reinforcement Learning of Koopman eNMPC](https://mlsanity.com/paper/2503.18787/) (2025)
- [Dreaming Falcon: Physics-Informed Model-Based Reinforcement Learning for Quadcopters](https://mlsanity.com/paper/2511.18243/) (2025)
- [Integrating Lagrangian Neural Networks into the Dyna Framework for Reinforcement Learning](https://mlsanity.com/paper/2603.08468/) (2026)
- [Articulated-Body Dynamics Network: Dynamics-Grounded Prior for Robot Learning](https://mlsanity.com/paper/2603.19078/) (2026)
- [Physics-informed reinforcement learning via probabilistic co-adjustment functions](https://mlsanity.com/paper/2309.05404/) (2023)
- [Facilitating Reinforcement Learning for Process Control Using Transfer Learning: Overview and Perspectives](https://mlsanity.com/paper/2404.00247/) (2024)
- [Trajectory First: A Curriculum for Discovering Diverse Policies](https://mlsanity.com/paper/2506.01568/) (2025)
- [Physics-Informed Reward Machines](https://mlsanity.com/paper/2508.14093/) (2025)
- [Control of dynamical systems with neural networks](https://mlsanity.com/paper/2510.12810/) (2025)
- [Data-Driven Physics Embedded Dynamics with Predictive Control and Reinforcement Learning for Quadrupeds](https://mlsanity.com/paper/2603.14333/) (2026)
- [Diffusion Sequence Models for Generative In-Context Meta-Learning of Robot Dynamics](https://mlsanity.com/paper/2604.13366/) (2026)
- [Adaptive Reconstruction of Bosonic Quantum States](https://mlsanity.com/paper/2608.02049/) (2026)

## Abstract (arXiv)

We apply reinforcement learning (RL) to robotics tasks. One of the drawbacks of traditional RL algorithms has been their poor sample efficiency. One approach to improve the sample efficiency is model-based RL. In our model-based RL algorithm, we learn a model of the environment, essentially its transition dynamics and reward function, use it to generate imaginary trajectories and backpropagate through them to update the policy, exploiting the differentiability of the model. Intuitively, learning more accurate models should lead to better model-based RL performance. Recently, there has been growing interest in developing better deep neural network based dynamics models for physical systems, by utilizing the structure of the underlying physics. We focus on robotic systems undergoing rigid body motion without contacts. We compare two versions of our model-based RL algorithm, one which uses a standard deep neural network based dynamics model and the other which uses a much more accurate, physics-informed neural network based dynamics model. We show that, in model-based RL, model accuracy mainly matters in environments that are sensitive to initial conditions, where numerical errors accumulate fast. In these environments, the physics-informed version of our algorithm achieves significantly better average-return and sample efficiency. In environments that are not sensitive to initial conditions, both versions of our algorithm achieve similar average-return, while the physics-informed version achieves better sample efficiency. We also show that, in challenging environments, physics-informed model-based RL achieves better average-return than state-of-the-art model-free RL algorithms such as Soft Actor-Critic, as it computes the policy-gradient analytically, while the latter estimates it through sampling.