# Physics-Informed Model and Hybrid Planning for Efficient Dyna-Style Reinforcement Learning

arXiv: https://arxiv.org/abs/2407.02217 · HTML: https://mlsanity.com/paper/2407.02217/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 31 of its 39 references that are in the map.
- First posted on arXiv 2 Jul 2024.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 30 of 31)

- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020)
- [Deep Reinforcement Learning at the Edge of the Statistical Precipice](https://mlsanity.com/paper/2108.13264/) (2021)
- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [Lagrangian Neural Networks](https://mlsanity.com/paper/2003.04630/) (2020)
- [Temporal Difference Learning for Model Predictive Control](https://mlsanity.com/paper/2203.04955/) (2022)
- [Challenges of Real-World Reinforcement Learning](https://mlsanity.com/paper/1904.12901/) (2019)
- [Residual Reinforcement Learning for Robot Control](https://mlsanity.com/paper/1812.03201/) (2018)
- [Sample-Efficient Reinforcement Learning with Stochastic Ensemble Value Expansion](https://mlsanity.com/paper/1807.01675/) (2018)
- [Learning Continuous Control Policies by Stochastic Value Gradients](https://mlsanity.com/paper/1510.09142/) (2015)
- [TossingBot: Learning to Throw Arbitrary Objects with Residual Physics](https://mlsanity.com/paper/1903.11239/) (2019)
- [Augmenting Physical Models with Deep Networks for Complex Dynamics Forecasting](https://mlsanity.com/paper/2010.04456/) (2020)
- [Exploring Model-based Planning with Policy Networks](https://mlsanity.com/paper/1906.08649/) (2019)
- [Model-Augmented Actor-Critic: Backpropagating through Paths](https://mlsanity.com/paper/2005.08068/) (2020)
- [Learning Off-Policy with Online Planning](https://mlsanity.com/paper/2008.10066/) (2020)
- [Continuous-Time Model-Based Reinforcement Learning](https://mlsanity.com/paper/2102.04764/) (2021)
- [Augmenting Physical Simulators with Stochastic Neural Networks: Case Study of Planar Pushing and Bouncing](https://mlsanity.com/paper/1808.03246/) (2018)
- [Blending MPC & Value Function Approximation for Efficient Reinforcement Learning](https://mlsanity.com/paper/2012.05909/) (2020)
- [Model Predictive Actor-Critic: Accelerating Robot Skill Acquisition with Deep Reinforcement Learning](https://mlsanity.com/paper/2103.13842/) (2021)
- [Physics-Informed Model-Based Reinforcement Learning](https://mlsanity.com/paper/2212.02179/) (2022)
- [The Benefits of Model-Based Generalization in Reinforcement Learning](https://mlsanity.com/paper/2211.02222/) (2022)
- [Evaluating model-based planning and planner amortization for continuous control](https://mlsanity.com/paper/2110.03363/) (2021)
- [Combining Learned and Analytical Models for Predicting Action Effects from Sensory Data](https://mlsanity.com/paper/1710.04102/) (2017)
- [Model-based Reinforcement Learning with Parametrized Physical Models and Optimism-Driven Exploration](https://mlsanity.com/paper/1509.06824/) (2015)

## Built on by (most cited first, 2 of 2)

- [RT-HCP: Dealing with Inference Delays and Sample Efficiency to Learn Directly on Robotic Platforms](https://mlsanity.com/paper/2509.06714/) (2025)
- [Sensory-Motor Control with Large Language Models via Iterative Policy Refinement](https://mlsanity.com/paper/2506.04867/) (2025)

## Abstract (arXiv)

Applying reinforcement learning (RL) to real-world applications requires addressing a trade-off between asymptotic performance, sample efficiency, and inference time. In this work, we demonstrate how to address this triple challenge by leveraging partial physical knowledge about the system dynamics. Our approach involves learning a physics-informed model to boost sample efficiency and generating imaginary trajectories from this model to learn a model-free policy and Q-function. Furthermore, we propose a hybrid planning strategy, combining the learned policy and Q-function with the learned model to enhance time efficiency in planning. Through practical demonstrations, we illustrate that our method improves the compromise between sample efficiency, time efficiency, and performance over state-of-the-art methods.