# Model-Based Reinforcement Learning for Control of Strongly-Disturbed Unsteady Aerodynamic Flows

arXiv: https://arxiv.org/abs/2408.14685 · HTML: https://mlsanity.com/paper/2408.14685/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in RL with diffusion and flow models, one of 15 directions in Deep reinforcement learning methods.
- Builds on 7 of its 21 references that are in the map.
- First posted on arXiv 26 Aug 2024.

## Placed in

- Deep reinforcement learning methods / [RL with diffusion and flow models](https://mlsanity.com/deeprl/rl-with-diffusion-and-flow-models/)

## Builds on (most cited first, 7 of 7)

- [Machine Learning for Fluid Mechanics](https://mlsanity.com/paper/1905.11075/) (2019)
- [Backpropagation Algorithms and Reservoir Computing in Recurrent Neural Networks for the Forecasting of Complex Spatiotemporal Dynamics](https://mlsanity.com/paper/1910.05266/) (2019)
- [State Representation Learning for Control: An Overview](https://mlsanity.com/paper/1802.04181/) (2018)
- [Efficient collective swimming by harnessing vortices through deep reinforcement learning](https://mlsanity.com/paper/1802.02674/) (2018)
- [Stabilized Neural Ordinary Differential Equations for Long-Time Forecasting of Dynamical Systems](https://mlsanity.com/paper/2203.15706/) (2022)
- [Grasping Extreme Aerodynamics on a Low-Dimensional Manifold](https://mlsanity.com/paper/2305.08024/) (2023)
- [Turbulence control in plane Couette flow using low-dimensional neural ODE-based models and deep reinforcement learning](https://mlsanity.com/paper/2301.12098/) (2023)

## Built on by (most cited first, 2 of 2)

- [Attention on flow control: transformer-based reinforcement learning for lift regulation in highly disturbed flows](https://mlsanity.com/paper/2506.10153/) (2025)
- [Statistical Turbulence and High-Fidelity Disturbance Fields for Quadrotor Flight Control](https://mlsanity.com/paper/2610.06874/) (2026)

## Abstract (arXiv)

The intrinsic high dimension of fluid dynamics is an inherent challenge to control of aerodynamic flows, and this is further complicated by a flow's nonlinear response to strong disturbances. Deep reinforcement learning, which takes advantage of the exploratory aspects of reinforcement learning (RL) and the rich nonlinearity of a deep neural network, provides a promising approach to discover feasible control strategies. However, the typical model-free approach to reinforcement learning requires a significant amount of interaction between the flow environment and the RL agent during training, and this high training cost impedes its development and application. In this work, we propose a model-based reinforcement learning (MBRL) approach by incorporating a novel reduced-order model as a surrogate for the full environment. The model consists of a physics-augmented autoencoder, which compresses high-dimensional CFD flow field snaphsots into a three-dimensional latent space, and a latent dynamics model that is trained to accurately predict the long-time dynamics of trajectories in the latent space in response to action sequences. The accuracy and robustness of the model are demonstrated in the scenario of a pitching airfoil within a highly disturbed environment. Additionally, an application to a vertical-axis wind turbine in a disturbance-free environment is discussed in the Appendix Based on the model trained in the pitching airfoil problem, we realize an MBRL strategy to mitigate lift variation during gust-airfoil encounters. We demonstrate that the policy learned in the reduced-order environment translates to an effective control strategy in the full CFD environment.