# Emergent behavior and neural dynamics in artificial agents tracking turbulent plumes

arXiv: https://arxiv.org/abs/2109.12434 · HTML: https://mlsanity.com/paper/2109.12434/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 126 references that are in the map.
- First posted on arXiv 25 Sep 2021.

## Builds on (most cited first, 23 of 23)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [High-Dimensional Continuous Control Using Generalized Advantage Estimation](https://mlsanity.com/paper/1506.02438/) (2015)
- [A Brief Survey of Deep Reinforcement Learning](https://mlsanity.com/paper/1708.05866/) (2017)
- [Multi-Task Learning with Deep Neural Networks: A Survey](https://mlsanity.com/paper/2009.09796/) (2020)
- [Embodied Intelligence via Learning and Evolution](https://mlsanity.com/paper/2102.02202/) (2021)
- [Efficient collective swimming by harnessing vortices through deep reinforcement learning](https://mlsanity.com/paper/1802.02674/) (2018)
- [Universality and individuality in neural dynamics across large populations of recurrent networks](https://mlsanity.com/paper/1907.08549/) (2019)
- [Emergence of grid-like representations by training recurrent neural networks to perform spatial localization](https://mlsanity.com/paper/1803.07770/) (2018)
- [Reverse engineering recurrent networks for sentiment classification reveals line attractor dynamics](https://mlsanity.com/paper/1906.10720/) (2019)
- [Recurrent Orthogonal Networks and Long-Memory Tasks](https://mlsanity.com/paper/1602.06662/) (2016)
- [Meta-trained agents implement Bayes-optimal agents](https://mlsanity.com/paper/2010.11223/) (2020)
- [Flow Navigation by Smart Microswimmers via Reinforcement Learning](https://mlsanity.com/paper/1701.08848/) (2017)
- [Deep Reinforcement Learning and its Neuroscientific Implications](https://mlsanity.com/paper/2007.03750/) (2020)
- [Neural tuning and representational geometry](https://mlsanity.com/paper/2104.09743/) (2021)
- [Deep neuroethology of a virtual rodent](https://mlsanity.com/paper/1911.09451/) (2019)
- [Inverse Rational Control with Partially Observable Continuous Nonlinear Dynamics](https://mlsanity.com/paper/1908.04696/) (2019)
- [Emergence of functional and structural properties of the head direction system by optimization of recurrent neural networks](https://mlsanity.com/paper/1912.10189/) (2019)
- [The population doctrine in cognitive neuroscience](https://mlsanity.com/paper/2104.00145/) (2021)
- [Predictability and hierarchy in Drosophila behavior](https://mlsanity.com/paper/1605.03626/) (2016)
- [Structure Learning in Motor Control:A Deep Reinforcement Learning Model](https://mlsanity.com/paper/1706.06827/) (2017)
- [Biological Mechanisms for Learning: A Computational Model of Olfactory Learning in the Manduca sexta Moth, with Applications to Neural Nets](https://mlsanity.com/paper/1802.02678/) (2018)
- [Training recurrent networks to generate hypotheses about how the brain solves hard navigation problems](https://mlsanity.com/paper/1609.09059/) (2016)

## Built on by (most cited first, 2 of 2)

- [From Data-Fitting to Discovery: Interpreting the Neural Dynamics of Motor Control through Reinforcement Learning](https://mlsanity.com/paper/2305.11107/) (2023)
- [A Population-Level Analysis of Neural Dynamics in Robust Legged Robots](https://mlsanity.com/paper/2306.15793/) (2023)

## Abstract (arXiv)

Tracking a turbulent plume to locate its source is a complex control problem because it requires multi-sensory integration and must be robust to intermittent odors, changing wind direction, and variable plume statistics. This task is routinely performed by flying insects, often over long distances, in pursuit of food or mates. Several aspects of this remarkable behavior have been studied in detail in many experimental studies. Here, we take a complementary in silico approach, using artificial agents trained with reinforcement learning to develop an integrated understanding of the behaviors and neural computations that support plume tracking. Specifically, we use deep reinforcement learning (DRL) to train recurrent neural network (RNN) agents to locate the source of simulated turbulent plumes. Interestingly, the agents' emergent behaviors resemble those of flying insects, and the RNNs learn to represent task-relevant variables, such as head direction and time since last odor encounter. Our analyses suggest an intriguing experimentally testable hypothesis for tracking plumes in changing wind direction -- that agents follow local plume shape rather than the current wind direction. While reflexive short-memory behaviors are sufficient for tracking plumes in constant wind, longer timescales of memory are essential for tracking plumes that switch direction. At the level of neural dynamics, the RNNs' population activity is low-dimensional and organized into distinct dynamical structures, with some correspondence to behavioral modules. Our in silico approach provides key intuitions for turbulent plume tracking strategies and motivates future targeted experimental and theoretical developments.