# Reinforcement learning in densely recurrent biological networks

arXiv: https://arxiv.org/abs/2508.09618 · HTML: https://mlsanity.com/paper/2508.09618/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 9 of its 45 references that are in the map.
- First posted on arXiv 13 Aug 2025.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 9 of 9)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Learning Sparse Neural Networks through L0 Regularization](https://mlsanity.com/paper/1712.01312/) (2017)
- [Evolution Strategies as a Scalable Alternative to Reinforcement Learning](https://mlsanity.com/paper/1703.03864/) (2017)
- [Ray: A Distributed Framework for Emerging AI Applications](https://mlsanity.com/paper/1712.05889/) (2017)
- [The CMA Evolution Strategy: A Tutorial](https://mlsanity.com/paper/1604.00772/) (2016)
- [Evolutionary Reinforcement Learning: A Survey](https://mlsanity.com/paper/2303.04150/) (2023)
- [A Computationally Efficient Limited Memory CMA-ES for Large Scale Optimization](https://mlsanity.com/paper/1404.5520/) (2014)
- [Joint optimization of well placement and control for nonconventional well types](https://mlsanity.com/paper/1409.4369/) (2014)
- [Soft-wired long-term memory in a natural recurrent neuronal network](https://mlsanity.com/paper/2004.00498/) (2020)

## Abstract (arXiv)

Training highly recurrent networks in continuous action spaces is a technical challenge: gradient-based methods suffer from exploding or vanishing gradients, while purely evolutionary searches converge slowly in high-dimensional weight spaces. We introduce a hybrid, derivative-free optimization framework that implements reinforcement learning by coupling global evolutionary exploration with local direct search exploitation. The method, termed ENOMAD (Evolutionary Nonlinear Optimization with Mesh Adaptive Direct search), is benchmarked on a suite of food-foraging tasks instantiated in the fully mapped neural connectome of the nematode Caenorhabditis elegans. Crucially, ENOMAD leverages biologically derived weight priors, letting it refine--rather than rebuild--the organism's native circuitry. Two algorithmic variants of the method are introduced, which lead to either small distributed adjustments of many weights, or larger changes on a limited number of weights. Both variants significantly exceed the performance of the untrained connectome (in what can be interpreted as an example of transfer learning) and of existing training strategies. These findings demonstrate that integrating evolutionary search with nonlinear optimization provides an efficient, biologically grounded strategy for specializing natural recurrent networks towards a specified set of tasks.