# Effective control of two-dimensional Rayleigh--B'enard convection: invariant multi-agent reinforcement learning is all you need

arXiv: https://arxiv.org/abs/2304.02370 · HTML: https://mlsanity.com/paper/2304.02370/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 10 of its 90 references that are in the map.
- First posted on arXiv 5 Apr 2023.

## Builds on (most cited first, 10 of 10)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Deep Reinforcement Learning: An Overview](https://mlsanity.com/paper/1701.07274/) (2017)
- [Tianshou: a Highly Modularized Deep Reinforcement Learning Library](https://mlsanity.com/paper/2107.14171/) (2021)
- [A review on Deep Reinforcement Learning for Fluid Mechanics](https://mlsanity.com/paper/1908.04127/) (2019)
- [Control of chaotic systems by Deep Reinforcement Learning](https://mlsanity.com/paper/1906.07672/) (2019)
- [Controlling Rayleigh-B'enard convection via Reinforcement Learning](https://mlsanity.com/paper/2003.14358/) (2020)
- [Deep Reinforcement Learning for Computational Fluid Dynamics on HPC Systems](https://mlsanity.com/paper/2205.06502/) (2022)
- [Comparative analysis of machine learning methods for active flow control](https://mlsanity.com/paper/2202.11664/) (2022)

## Built on by (most cited first, 27 of 27)

- [Multi-agent reinforcement learning for the control of three-dimensional Rayleigh-B'enard convection](https://mlsanity.com/paper/2407.21565/) (2024)
- [Equivariance and partial observations in Koopman operator theory for partial differential equations](https://mlsanity.com/paper/2307.15325/) (2023)
- [Advanced deep-reinforcement-learning methods for flow control: group-invariant and positional-encoding networks improve learning speed and quality](https://mlsanity.com/paper/2407.17822/) (2024)
- [Navigation in a simplified Urban Flow through Deep Reinforcement Learning](https://mlsanity.com/paper/2409.17922/) (2024)
- [Model-based deep reinforcement learning for accelerated learning from flow simulations](https://mlsanity.com/paper/2402.16543/) (2024)
- [Real-time optimal control of high-dimensional parametrized systems by deep learning-based reduced order models](https://mlsanity.com/paper/2409.05709/) (2024)
- [Active flow control for three-dimensional cylinders through deep reinforcement learning](https://mlsanity.com/paper/2309.02462/) (2023)
- [Koopman-Based Surrogate Modelling of Turbulent Rayleigh-B'enard Convection](https://mlsanity.com/paper/2405.06425/) (2024)
- [Surrogate Modeling of 3D Rayleigh-Benard Convection with Equivariant Autoencoders](https://mlsanity.com/paper/2505.13569/) (2025)
- [Gym-preCICE: Reinforcement Learning Environments for Active Flow Control](https://mlsanity.com/paper/2305.02033/) (2023)
- [Solving Partial Differential Equations with Equivariant Extreme Learning Machines](https://mlsanity.com/paper/2404.18530/) (2024)
- [Multi-fidelity Reinforcement Learning Control for Complex Dynamical Systems](https://mlsanity.com/paper/2504.05588/) (2025)
- [Control of Rayleigh-B'enard Convection: Effectiveness of Reinforcement Learning in the Turbulent Regime](https://mlsanity.com/paper/2504.12000/) (2025)
- [Plug-and-Play Benchmarking of Reinforcement Learning Algorithms for Large-Scale Flow Control](https://mlsanity.com/paper/2601.15015/) (2026)
- [Reinforcement Learning for Freeway Lane-Change Regulation via Connected Vehicles](https://mlsanity.com/paper/2412.04341/) (2024)
- [Understanding the role of autoencoders for stiff dynamical systems using information theory](https://mlsanity.com/paper/2503.06325/) (2025)
- [Invariant Control Strategies for Active Flow Control using Graph Neural Networks](https://mlsanity.com/paper/2503.22775/) (2025)
- [HypeMARL: Multi-Agent Reinforcement Learning For High-Dimensional, Parametric, and Distributed Systems](https://mlsanity.com/paper/2509.16709/) (2025)
- [Reward hacking in physical reinforcement learning revealed by turbulent drag reduction](https://mlsanity.com/paper/2606.06227/) (2026)
- [Latent feedback control of distributed systems in multiple scenarios through deep learning-based reduced order models](https://mlsanity.com/paper/2412.09942/) (2024)
- [Koopman-based surrogate modeling for reinforcement-learning-control of Rayleigh-Benard convection](https://mlsanity.com/paper/2603.28074/) (2026)
- [Hyperfastrl: Hypernetwork-based reinforcement learning for unified control of parametric chaotic PDEs](https://mlsanity.com/paper/2604.06497/) (2026)
- [Timescale Separation Enables Deep Reinforcement Learning Control of Rotating Detonation Engine Mode Transitions](https://mlsanity.com/paper/2604.14398/) (2026)
- [Explainable deep reinforcement learning reveals energy-efficient control strategies for turbulent drag reduction](https://mlsanity.com/paper/2606.00949/) (2026)
- [Offline Reinforcement Learning for Fluid Controls: Data-based Multi-observational Policy Extraction](https://mlsanity.com/paper/2606.31025/) (2026)
- [Gradient-free learning of a closed-loop wall controller for turbulent drag reduction](https://mlsanity.com/paper/2607.12626/) (2026)
- [Physics-enhanced reinforcement learning for real-time optimal control of dynamical systems](https://mlsanity.com/paper/2607.16177/) (2026)

## Abstract (arXiv)

Rayleigh-B'enard convection (RBC) is a recurrent phenomenon in several industrial and geoscience flows and a well-studied system from a fundamental fluid-mechanics viewpoint. However, controlling RBC, for example by modulating the spatial distribution of the bottom-plate heating in the canonical RBC configuration, remains a challenging topic for classical control-theory methods. In the present work, we apply deep reinforcement learning (DRL) for controlling RBC. We show that effective RBC control can be obtained by leveraging invariant multi-agent reinforcement learning (MARL), which takes advantage of the locality and translational invariance inherent to RBC flows inside wide channels. The MARL framework applied to RBC allows for an increase in the number of control segments without encountering the curse of dimensionality that would result from a naive increase in the DRL action-size dimension. This is made possible by the MARL ability for re-using the knowledge generated in different parts of the RBC domain. We show in a case study that MARL DRL is able to discover an advanced control strategy that destabilizes the spontaneous RBC double-cell pattern, changes the topology of RBC by coalescing adjacent convection cells, and actively controls the resulting coalesced cell to bring it to a new stable configuration. This modified flow configuration results in reduced convective heat transfer, which is beneficial in several industrial processes. Therefore, our work both shows the potential of MARL DRL for controlling large RBC systems, as well as demonstrates the possibility for DRL to discover strategies that move the RBC configuration between different topological configurations, yielding desirable heat-transfer characteristics. These results are useful for both gaining further understanding of the intrinsic properties of RBC, as well as for developing industrial applications.