# Model-based deep reinforcement learning for accelerated learning from flow simulations

arXiv: https://arxiv.org/abs/2402.16543 · HTML: https://mlsanity.com/paper/2402.16543/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in RL with diffusion and flow models, one of 15 directions in Deep reinforcement learning methods.
- Builds on 8 of its 31 references that are in the map.
- First posted on arXiv 26 Feb 2024.

## Placed in

- Deep reinforcement learning methods / [RL with diffusion and flow models](https://mlsanity.com/deeprl/rl-with-diffusion-and-flow-models/)

## Builds on (most cited first, 8 of 8)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Model-Ensemble Trust-Region Policy Optimization](https://mlsanity.com/paper/1802.10592/) (2018)
- [Dynamic mode decomposition with control](https://mlsanity.com/paper/1409.6358/) (2014)
- [A review on Deep Reinforcement Learning for Fluid Mechanics](https://mlsanity.com/paper/1908.04127/) (2019)
- [Effective control of two-dimensional Rayleigh--B'enard convection: invariant multi-agent reinforcement learning is all you need](https://mlsanity.com/paper/2304.02370/) (2023)
- [PPO-CMA: Proximal Policy Optimization with Covariance Matrix Adaptation](https://mlsanity.com/paper/1810.02541/) (2018)
- [Deep Reinforcement Learning for Heat Pump Control](https://mlsanity.com/paper/2212.12716/) (2022)
- [Robust optimal well control using an adaptive multi-grid reinforcement learning framework](https://mlsanity.com/paper/2207.03253/) (2022)

## Built on by (most cited first, 4 of 4)

- [Invariant Control Strategies for Active Flow Control using Graph Neural Networks](https://mlsanity.com/paper/2503.22775/) (2025)
- [SALSA-RL: Stability Analysis in the Latent Space of Actions for Reinforcement Learning](https://mlsanity.com/paper/2502.15512/) (2025)
- [Enhancing sample efficiency in reinforcement-learning-based flow control: replacing the critic with an adaptive reduced-order model](https://mlsanity.com/paper/2604.04986/) (2026)
- [Physics-guided surrogate learning enables zero-shot control of turbulent wings](https://mlsanity.com/paper/2604.09434/) (2026)

## Abstract (arXiv)

In recent years, deep reinforcement learning has emerged as a technique to solve closed-loop flow control problems. Employing simulation-based environments in reinforcement learning enables a priori end-to-end optimization of the control system, provides a virtual testbed for safety-critical control applications, and allows to gain a deep understanding of the control mechanisms. While reinforcement learning has been applied successfully in a number of rather simple flow control benchmarks, a major bottleneck toward real-world applications is the high computational cost and turnaround time of flow simulations. In this contribution, we demonstrate the benefits of model-based reinforcement learning for flow control applications. Specifically, we optimize the policy by alternating between trajectories sampled from flow simulations and trajectories sampled from an ensemble of environment models. The model-based learning reduces the overall training time by up to 85% for the fluidic pinball test case. Even larger savings are expected for more demanding flow simulations.