# Q-learning with temporal memory to navigate turbulence

arXiv: https://arxiv.org/abs/2404.17495 · HTML: https://mlsanity.com/paper/2404.17495/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in RL for robot navigation and UAV control, one of 15 directions in Deep reinforcement learning methods.
- Builds on 2 of its 76 references that are in the map.
- First posted on arXiv 26 Apr 2024.

## Placed in

- Deep reinforcement learning methods / [RL for robot navigation and UAV control](https://mlsanity.com/deeprl/rl-for-robot-navigation-and-uav-control/)

## Builds on (most cited first, 2 of 2)

- [Deep reinforcement learning for the olfactory search POMDP: a quantitative benchmark](https://mlsanity.com/paper/2302.00706/) (2023)
- [Optimal policies for Bayesian olfactory search in turbulent flows](https://mlsanity.com/paper/2207.04277/) (2022)

## Built on by (most cited first, 5 of 5)

- [Controlling dynamics of stochastic systems with deep reinforcement learning](https://mlsanity.com/paper/2502.18111/) (2025)
- [A critical assessment of reinforcement learning methods for microswimmer navigation in complex flows](https://mlsanity.com/paper/2505.05525/) (2025)
- [Clock-state olfactory search in turbulent flows using Q-learning: The geometry of plume recovery](https://mlsanity.com/paper/2605.15938/) (2026)
- [Emergence of a Flow-Assisted Casting Strategy for Olfactory Navigation via Memory-Augmented Reinforcement Learning](https://mlsanity.com/paper/2605.18881/) (2026)
- [Information-Guided Safe Reinforcement Learning for Autonomous Gas Source Localization using sUAS](https://mlsanity.com/paper/2609.05569/) (2026)

## Abstract (arXiv)

We consider the problem of olfactory searches in a turbulent environment. We focus on agents that respond solely to odor stimuli, with no access to spatial perception nor prior information about the odor. We ask whether navigation to a target can be learned robustly within a sequential decision making framework. We develop a reinforcement learning algorithm using a small set of interpretable olfactory states and train it with realistic turbulent odor cues. By introducing a temporal memory, we demonstrate that two salient features of odor traces, discretized in few olfactory states, are sufficient to learn navigation in a realistic odor plume. Performance is dictated by the sparse nature of turbulent odors. An optimal memory exists which ignores blanks within the plume and activates a recovery strategy outside the plume. We obtain the best performance by letting agents learn their recovery strategy and show that it is mostly casting cross wind, similar to behavior observed in flying insects. The optimal strategy is robust to substantial changes in the odor plumes, suggesting minor parameter tuning may be sufficient to adapt to different environments.