# A critical assessment of reinforcement learning methods for microswimmer navigation in complex flows

arXiv: https://arxiv.org/abs/2505.05525 · HTML: https://mlsanity.com/paper/2505.05525/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for robot navigation and UAV control, one of 15 directions in Deep reinforcement learning methods.
- Builds on 18 of its 50 references that are in the map.
- First posted on arXiv 8 May 2025.

## Placed in

- Deep reinforcement learning methods / [RL for robot navigation and UAV control](https://mlsanity.com/deeprl/rl-for-robot-navigation-and-uav-control/)

## Builds on (most cited first, 18 of 18)

- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study](https://mlsanity.com/paper/2006.05990/) (2020)
- [Efficient collective swimming by harnessing vortices through deep reinforcement learning](https://mlsanity.com/paper/1802.02674/) (2018)
- [Flow Navigation by Smart Microswimmers via Reinforcement Learning](https://mlsanity.com/paper/1701.08848/) (2017)
- [Learning Efficient Navigation in Vortical Flow Fields](https://mlsanity.com/paper/2102.10536/) (2021)
- [Zermelo's problem: Optimal point-to-point navigation in 2D turbulent flows using Reinforcement Learning](https://mlsanity.com/paper/1907.08591/) (2019)
- [Machine learning strategies for path-planning microswimmers in turbulent flows](https://mlsanity.com/paper/1910.01728/) (2019)
- [Finding Efficient Swimming Strategies in a Three Dimensional Chaotic Flow by Reinforcement Learning](https://mlsanity.com/paper/1711.05826/) (2017)
- [Deep reinforcement learning for the olfactory search POMDP: a quantitative benchmark](https://mlsanity.com/paper/2302.00706/) (2023)
- [Optimal tracking strategies in a turbulent flow](https://mlsanity.com/paper/2305.04677/) (2023)
- [Q-learning with temporal memory to navigate turbulence](https://mlsanity.com/paper/2404.17495/) (2024)
- [Learning swimming escape patterns for larval fish under energy constraints](https://mlsanity.com/paper/2105.00771/) (2021)
- [Taming Lagrangian Chaos with Multi-Objective Reinforcement Learning](https://mlsanity.com/paper/2212.09612/) (2022)
- [Learning to swim efficiently in a nonuniform flow field](https://mlsanity.com/paper/2212.11482/) (2022)
- [Fish-inspired tracking of underwater turbulent plumes](https://mlsanity.com/paper/2403.06091/) (2024)

## Abstract (arXiv)

Navigating in a fluid flow while being carried by it, using only information accessible from on-board sensors, is a problem commonly faced by small planktonic organisms. It is also directly relevant to autonomous robots deployed in the oceans. In the last ten years, the fluid mechanics community has widely adopted reinforcement learning, often in the form of its simplest implementations, to address this challenge. But it is unclear how good are the strategies learned by these algorithms. In this paper, we perform a quantitative assessment of reinforcement learning methods applied to navigation in partially observable flows. We first introduce a well-posed problem of directional navigation for which a quasi-optimal policy is known analytically. We then report on the poor performance and robustness of commonly used algorithms (Q-Learning, Advantage Actor Critic) in flows regularly encountered in the literature: Taylor-Green vortices, Arnold-Beltrami-Childress flow, and two-dimensional turbulence. We show that they are vastly surpassed by PPO (Proximal Policy Optimization), a more advanced algorithm that has established dominance across a wide range of benchmarks in the reinforcement learning community. In particular, our custom implementation of PPO matches the theoretical quasi-optimal performance in turbulent flow and does so in a robust manner. Reaching this result required the use of several additional techniques, such as vectorized environments and generalized advantage estimation, as well as hyperparameter optimization. This study demonstrates the importance of algorithm selection, implementation details, and fine-tuning for discovering truly smart autonomous navigation strategies in complex flows.