# Physics-informed Dyna-Style Model-Based Deep Reinforcement Learning for Dynamic Control

arXiv: https://arxiv.org/abs/2108.00128 · HTML: https://mlsanity.com/paper/2108.00128/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 74 references that are in the map.
- First posted on arXiv 31 Jul 2021.

## Builds on (most cited first, 30 of 39)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challenges](https://mlsanity.com/paper/2011.06225/) (2020)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [Dueling Network Architectures for Deep Reinforcement Learning](https://mlsanity.com/paper/1511.06581/) (2015)
- [When to Trust Your Model: Model-Based Policy Optimization](https://mlsanity.com/paper/1906.08253/) (2019)
- [Model-Based Reinforcement Learning for Atari](https://mlsanity.com/paper/1903.00374/) (2019)
- [SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives](https://mlsanity.com/paper/1407.0202/) (2014)
- [Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning](https://mlsanity.com/paper/1708.02596/) (2017)
- [Model-Ensemble Trust-Region Policy Optimization](https://mlsanity.com/paper/1802.10592/) (2018)
- [Deep Dynamics Models for Learning Dexterous Manipulation](https://mlsanity.com/paper/1909.11652/) (2019)
- [PhyGeoNet: Physics-Informed Geometry-Adaptive Convolutional Neural Networks for Solving Parameterized Steady-State PDEs on Irregular Domain](https://mlsanity.com/paper/2004.13145/) (2020)
- [Physics-informed learning of governing equations from scarce data](https://mlsanity.com/paper/2005.03448/) (2020)
- [Minimizing Finite Sums with the Stochastic Average Gradient](https://mlsanity.com/paper/1309.2388/) (2013)
- [Learning Continuous Control Policies by Stochastic Value Gradients](https://mlsanity.com/paper/1510.09142/) (2015)
- [Benchmarking Model-Based Reinforcement Learning](https://mlsanity.com/paper/1907.02057/) (2019)
- [Imagination-Augmented Agents for Deep Reinforcement Learning](https://mlsanity.com/paper/1707.06203/) (2017)
- [Algorithmic Framework for Model-based Deep Reinforcement Learning with Theoretical Guarantees](https://mlsanity.com/paper/1807.03858/) (2018)
- [Physics-informed graph neural Galerkin networks: A unified framework for solving PDE-governed forward and inverse problems](https://mlsanity.com/paper/2107.12146/) (2021)
- [Control of a Quadrotor with Reinforcement Learning](https://mlsanity.com/paper/1707.05110/) (2017)
- [Physics informed deep learning for computational elastodynamics without labeled data](https://mlsanity.com/paper/2006.08472/) (2020)
- [Efficient collective swimming by harnessing vortices through deep reinforcement learning](https://mlsanity.com/paper/1802.02674/) (2018)
- [A review on Deep Reinforcement Learning for Fluid Mechanics](https://mlsanity.com/paper/1908.04127/) (2019)
- [Learning and Policy Search in Stochastic Dynamical Systems with Bayesian Neural Networks](https://mlsanity.com/paper/1605.07127/) (2016)

## Built on by (most cited first, 23 of 23)

- [Distributed Control of Partial Differential Equations Using Convolutional Reinforcement Learning](https://mlsanity.com/paper/2301.10737/) (2023)
- [A Survey on Physics Informed Reinforcement Learning: Review and Open Problems](https://mlsanity.com/paper/2309.01909/) (2023)
- [Deep Learning in Deterministic Computational Mechanics](https://mlsanity.com/paper/2309.15421/) (2023)
- [Data-driven control of spatiotemporal chaos with reduced-order neural ODE-based models and reinforcement learning](https://mlsanity.com/paper/2205.00579/) (2022)
- [Preconditioning for Physics-Informed Neural Networks](https://mlsanity.com/paper/2402.00531/) (2024)
- [Digital Twin-Enhanced Wireless Indoor Navigation: Achieving Efficient Environment Sensing with Zero-Shot Reinforcement Learning](https://mlsanity.com/paper/2306.06766/) (2023)
- [How to Learn and Generalize From Three Minutes of Data: Physics-Constrained and Uncertainty-Aware Neural Stochastic Differential Equations](https://mlsanity.com/paper/2306.06335/) (2023)
- [A Mass-Conserving-Perceptron for Machine Learning-Based Modeling of Geoscientific Systems](https://mlsanity.com/paper/2310.08644/) (2023)
- [Learning a model is paramount for sample efficiency in reinforcement learning control of PDEs](https://mlsanity.com/paper/2302.07160/) (2023)
- [Controlgym: Large-Scale Control Environments for Benchmarking Reinforcement Learning Algorithms](https://mlsanity.com/paper/2311.18736/) (2023)
- [Physics-Informed Neural Networks with Skip Connections for Modeling and Control of Gas-Lifted Oil Wells](https://mlsanity.com/paper/2403.02289/) (2024)
- [Sample-Efficient Reinforcement Learning of Koopman eNMPC](https://mlsanity.com/paper/2503.18787/) (2025)
- [Multi-fidelity Reinforcement Learning Control for Complex Dynamical Systems](https://mlsanity.com/paper/2504.05588/) (2025)
- [Asynchronous Parallel Reinforcement Learning for Optimizing Propulsive Performance in Fin Ray Control](https://mlsanity.com/paper/2401.11349/) (2024)
- [Data-assimilated model-informed reinforcement learning](https://mlsanity.com/paper/2506.01755/) (2025)
- [Integrating Lagrangian Neural Networks into the Dyna Framework for Reinforcement Learning](https://mlsanity.com/paper/2603.08468/) (2026)
- [How to Re-enable PDE Loss for Physical Systems Modeling Under Partial Observation](https://mlsanity.com/paper/2412.09116/) (2024)
- [SALSA-RL: Stability Analysis in the Latent Space of Actions for Reinforcement Learning](https://mlsanity.com/paper/2502.15512/) (2025)
- [Reinforcement Twinning for Hybrid Control of Flapping-Wing Drones](https://mlsanity.com/paper/2505.18201/) (2025)
- [Hyperfastrl: Hypernetwork-based reinforcement learning for unified control of parametric chaotic PDEs](https://mlsanity.com/paper/2604.06497/) (2026)
- [Physics-Informed Deep Learning for Entropy Prediction in Heterogeneous Systems: Thermodynamic and Information-Theoretic Case Studies](https://mlsanity.com/paper/2606.01179/) (2026)
- [Offline Reinforcement Learning for Fluid Controls: Data-based Multi-observational Policy Extraction](https://mlsanity.com/paper/2606.31025/) (2026)
- [Physics-enhanced reinforcement learning for real-time optimal control of dynamical systems](https://mlsanity.com/paper/2607.16177/) (2026)

## Abstract (arXiv)

Model-based reinforcement learning (MBRL) is believed to have much higher sample efficiency compared to model-free algorithms by learning a predictive model of the environment. However, the performance of MBRL highly relies on the quality of the learned model, which is usually built in a black-box manner and may have poor predictive accuracy outside of the data distribution. The deficiencies of the learned model may prevent the policy from being fully optimized. Although some uncertainty analysis-based remedies have been proposed to alleviate this issue, model bias still poses a great challenge for MBRL. In this work, we propose to leverage the prior knowledge of underlying physics of the environment, where the governing laws are (partially) known. In particular, we developed a physics-informed MBRL framework, where governing equations and physical constraints are utilized to inform the model learning and policy search. By incorporating the prior information of the environment, the quality of the learned model can be notably improved, while the required interactions with the environment are significantly reduced, leading to better sample efficiency and learning performance. The effectiveness and merit have been demonstrated over a handful of classic control problems, where the environments are governed by canonical ordinary/partial differential equations.