# Data-driven control of spatiotemporal chaos with reduced-order neural ODE-based models and reinforcement learning

arXiv: https://arxiv.org/abs/2205.00579 · HTML: https://mlsanity.com/paper/2205.00579/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 67 references that are in the map.
- First posted on arXiv 1 May 2022.

## Builds on (most cited first, 25 of 25)

- [Neural Ordinary Differential Equations](https://mlsanity.com/paper/1806.07366/) (2018)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [Recurrent World Models Facilitate Policy Evolution](https://mlsanity.com/paper/1809.01999/) (2018)
- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [Data-Efficient Hierarchical Reinforcement Learning](https://mlsanity.com/paper/1805.08296/) (2018)
- [Deep Reinforcement Learning: An Overview](https://mlsanity.com/paper/1701.07274/) (2017)
- [Neural Network Dynamics for Model-Based Deep Reinforcement Learning with Model-Free Fine-Tuning](https://mlsanity.com/paper/1708.02596/) (2017)
- [Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Images](https://mlsanity.com/paper/1506.07365/) (2015)
- [Model-Based Value Estimation for Efficient Model-Free Reinforcement Learning](https://mlsanity.com/paper/1803.00101/) (2018)
- [Is a Good Representation Sufficient for Sample Efficient Reinforcement Learning?](https://mlsanity.com/paper/1910.03016/) (2019)
- [Backpropagation Algorithms and Reservoir Computing in Recurrent Neural Networks for the Forecasting of Complex Spatiotemporal Dynamics](https://mlsanity.com/paper/1910.05266/) (2019)
- [Dynamic mode decomposition with control](https://mlsanity.com/paper/1409.6358/) (2014)
- [Multiscale Simulations of Complex Systems by Learning their Effective Dynamics](https://mlsanity.com/paper/2006.13431/) (2020)
- [From Pixels to Torques: Policy Learning with Deep Dynamical Models](https://mlsanity.com/paper/1502.02251/) (2015)
- [Data-driven discovery of intrinsic dynamics](https://mlsanity.com/paper/2108.05928/) (2021)
- [Towards extraction of orthogonal and parsimonious non-linear modes from turbulent flows](https://mlsanity.com/paper/2109.01514/) (2021)
- [Deep learning to discover and predict dynamics on an inertial manifold](https://mlsanity.com/paper/2001.04263/) (2019)
- [Data-Driven Reduced-Order Modeling of Spatiotemporal Chaos with Neural Ordinary Differential Equations](https://mlsanity.com/paper/2109.00060/) (2021)
- [Control of chaotic systems by Deep Reinforcement Learning](https://mlsanity.com/paper/1906.07672/) (2019)
- [Controlling Rayleigh-B'enard convection via Reinforcement Learning](https://mlsanity.com/paper/2003.14358/) (2020)
- [Physics-informed Dyna-Style Model-Based Deep Reinforcement Learning for Dynamic Control](https://mlsanity.com/paper/2108.00128/) (2021)
- [Symmetry reduction for deep reinforcement learning active control of chaotic spatiotemporal dynamics](https://mlsanity.com/paper/2104.05437/) (2021)

## Built on by (most cited first, 17 of 17)

- [Data-driven Control of Agent-based Models: an Equation/Variable-free Machine Learning Approach](https://mlsanity.com/paper/2207.05779/) (2022)
- [Data-driven low-dimensional dynamic model of Kolmogorov flow](https://mlsanity.com/paper/2210.16708/) (2022)
- [Deep learning delay coordinate dynamics for chaotic attractors from partial observable data](https://mlsanity.com/paper/2211.11061/) (2022)
- [Dynamics of a data-driven low-dimensional model of turbulent minimal Couette flow](https://mlsanity.com/paper/2301.04638/) (2023)
- [Learning a model is paramount for sample efficiency in reinforcement learning control of PDEs](https://mlsanity.com/paper/2302.07160/) (2023)
- [Controlgym: Large-Scale Control Environments for Benchmarking Reinforcement Learning Algorithms](https://mlsanity.com/paper/2311.18736/) (2023)
- [Turbulence control in plane Couette flow using low-dimensional neural ODE-based models and deep reinforcement learning](https://mlsanity.com/paper/2301.12098/) (2023)
- [Real-time optimal control of high-dimensional parametrized systems by deep learning-based reduced order models](https://mlsanity.com/paper/2409.05709/) (2024)
- [Solving Partial Differential Equations with Equivariant Extreme Learning Machines](https://mlsanity.com/paper/2404.18530/) (2024)
- [Data-assimilated model-informed reinforcement learning](https://mlsanity.com/paper/2506.01755/) (2025)
- [Machine-Learned Closure of URANS for Stably Stratified Turbulence: Connecting Physical Timescales & Data Hyperparameters of Deep Time-Series Models](https://mlsanity.com/paper/2404.16141/) (2024)
- [Data-driven low-dimensional model of a sedimenting flexible fiber](https://mlsanity.com/paper/2405.10442/) (2024)
- [Latent feedback control of distributed systems in multiple scenarios through deep learning-based reduced order models](https://mlsanity.com/paper/2412.09942/) (2024)
- [Blending data and physics for reduced-order modeling of systems with spatiotemporal chaotic dynamics](https://mlsanity.com/paper/2507.21299/) (2025)
- [Transfer learning strategies for accelerating reinforcement-learning-based flow control](https://mlsanity.com/paper/2510.16016/) (2025)
- [Generative prediction of laser-induced rocket ignition with dynamic latent space representations](https://mlsanity.com/paper/2603.07525/) (2026)
- [Hyperfastrl: Hypernetwork-based reinforcement learning for unified control of parametric chaotic PDEs](https://mlsanity.com/paper/2604.06497/) (2026)

## Abstract (arXiv)

Deep reinforcement learning (RL) is a data-driven method capable of discovering complex control strategies for high-dimensional systems, making it promising for flow control applications. In particular, the present work is motivated by the goal of reducing energy dissipation in turbulent flows, and the example considered is the spatiotemporally chaotic dynamics of the Kuramoto-Sivashinsky equation (KSE). A major challenge associated with RL is that substantial training data must be generated by repeatedly interacting with the target system, making it costly when the system is computationally or experimentally expensive. We mitigate this challenge in a data-driven manner by combining dimensionality reduction via an autoencoder with a neural ODE framework to obtain a low-dimensional dynamical model from just a limited data set. We substitute this data-driven reduced-order model (ROM) in place of the true system during RL training to efficiently estimate the optimal policy, which can then be deployed on the true system. For the KSE actuated with localized forcing ("jets") at four locations, we demonstrate that we are able to learn a ROM that accurately captures the actuated dynamics as well as the underlying natural dynamics just from snapshots of the KSE experiencing random actuations. Using this ROM and a control objective of minimizing dissipation and power cost, we extract a control policy from it using deep RL. We show that the ROM-based control strategy translates well to the true KSE and highlight that the RL agent discovers and stabilizes an underlying forced equilibrium solution of the KSE system. We show that this forced equilibrium captured in the ROM and discovered through RL is related to an existing known equilibrium solution of the natural KSE.