# Gradient-free learning of a closed-loop wall controller for turbulent drag reduction

arXiv: https://arxiv.org/abs/2607.12626 · HTML: https://mlsanity.com/paper/2607.12626/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Multi-agent reinforcement learning, one of 15 directions in Deep reinforcement learning methods.
- Builds on 14 of its 58 references that are in the map.
- First posted on arXiv 14 Jul 2026.

## Placed in

- Deep reinforcement learning methods / [Multi-agent reinforcement learning](https://mlsanity.com/deeprl/multi-agent-reinforcement-learning/)

## Builds on (most cited first, 14 of 14)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [Counterfactual Multi-Agent Policy Gradients](https://mlsanity.com/paper/1705.08926/) (2017)
- [Sim-to-Real Transfer of Robotic Control with Dynamics Randomization](https://mlsanity.com/paper/1710.06537/) (2017)
- [Evolution Strategies as a Scalable Alternative to Reinforcement Learning](https://mlsanity.com/paper/1703.03864/) (2017)
- [Machine Learning for Fluid Mechanics](https://mlsanity.com/paper/1905.11075/) (2019)
- [Solver-in-the-Loop: Learning from Differentiable Physics to Interact with Iterative PDE-Solvers](https://mlsanity.com/paper/2007.00016/) (2020)
- [A review on Deep Reinforcement Learning for Fluid Mechanics](https://mlsanity.com/paper/1908.04127/) (2019)
- [Scientific multi-agent reinforcement learning for wall-models of turbulent flows](https://mlsanity.com/paper/2106.11144/) (2021)
- [Effective control of two-dimensional Rayleigh--B'enard convection: invariant multi-agent reinforcement learning is all you need](https://mlsanity.com/paper/2304.02370/) (2023)
- [GPU acceleration of CaNS for massively-parallel direct numerical simulations of canonical fluid flows](https://mlsanity.com/paper/2001.05234/) (2020)
- [Reward hacking in physical reinforcement learning revealed by turbulent drag reduction](https://mlsanity.com/paper/2606.06227/) (2026)

## Abstract (arXiv)

Closed-loop wall controllers learnt by multi-agent reinforcement learning are usually trained on periodic boxes far smaller than the flows they are meant to drive, and a large part of their drag reduction is lost when they are carried across. Retraining on the target domain is not an affordable remedy: the centralised critic that assigns credit to each wall patch degrades as patches are added, the zero-net-mass constraint couples the patches it is asked to separate, and the episodes must be collected in sequence at a cost that grows with the domain. We propose instead a short gradient-free refinement stage, applied to the transferred policy on the domain it will drive. An evolution strategy scores whole flow episodes against a regularised objective, so no credit has to be assigned to individual patches, and the candidates in a generation are independent and run in parallel. Applied to a recurrent multi-agent policy trained on a minimal flow unit at Retausimeq180 and evaluated on a channel sixteen times larger in wall-parallel area, a few generations raise the drag reduction from 19.0% to 25.7%, above the 22.5% of opposition control. Because the policies before and after refinement share an architecture and a training history, the difference between the controlled flows follows from the refinement alone. It shows in the friction decomposition, in the Reynolds stresses and in the near-wall spectra, and the actuation moves from a weak coupling to the wall-normal velocity towards a strong coupling to the streamwise fluctuation.