# Evolving Reservoirs for Meta Reinforcement Learning

arXiv: https://arxiv.org/abs/2312.06695 · HTML: https://mlsanity.com/paper/2312.06695/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 50 references that are in the map.
- First posted on arXiv 9 Dec 2023.

## Builds on (most cited first, 15 of 15)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Optuna: A Next-generation Hyperparameter Optimization Framework](https://mlsanity.com/paper/1907.10902/) (2019)
- [Dota 2 with Large Scale Deep Reinforcement Learning](https://mlsanity.com/paper/1912.06680/) (2019)
- [The CMA Evolution Strategy: A Tutorial](https://mlsanity.com/paper/1604.00772/) (2016)
- [Deep Reinforcement Learning: An Overview](https://mlsanity.com/paper/1701.07274/) (2017)
- [A Tutorial on Meta-Reinforcement Learning](https://mlsanity.com/paper/2301.08028/) (2023)
- [AI-GAs: AI-generating algorithms, an alternate paradigm for producing general artificial intelligence](https://mlsanity.com/paper/1905.10985/) (2019)
- [Towards Self-Assembling Artificial Neural Networks through Neural Developmental Programs](https://mlsanity.com/paper/2307.08197/) (2023)
- [Evolving and Merging Hebbian Learning Rules: Increasing Generalization by Decreasing the Number of Rules](https://mlsanity.com/paper/2104.07959/) (2021)
- [Reinforcement Learning with Convolutional Reservoir Computing](https://mlsanity.com/paper/1912.04161/) (2019)
- [Distributive Dynamic Spectrum Access through Deep Reinforcement Learning: A Reservoir Computing Based Approach](https://mlsanity.com/paper/1810.11758/) (2018)
- [Learning to Act through Evolution of Neural Diversity in Random Neural Networks](https://mlsanity.com/paper/2305.15945/) (2023)
- [Multiple chaotic central pattern generators with learning for legged locomotion and malfunction compensation](https://mlsanity.com/paper/1407.3269/) (2014)

## Built on by (most cited first, 1 of 1)

- [Analyzing the Dynamics of WSBM-based Echo State Networks using InputDSA](https://mlsanity.com/paper/2610.15380/) (2026)

## Abstract (arXiv)

Animals often demonstrate a remarkable ability to adapt to their environments during their lifetime. They do so partly due to the evolution of morphological and neural structures. These structures capture features of environments shared between generations to bias and speed up lifetime learning. In this work, we propose a computational model for studying a mechanism that can enable such a process. We adopt a computational framework based on meta reinforcement learning as a model of the interplay between evolution and development. At the evolutionary scale, we evolve reservoirs, a family of recurrent neural networks that differ from conventional networks in that one optimizes not the synaptic weights, but hyperparameters controlling macro-level properties of the resulting network architecture. At the developmental scale, we employ these evolved reservoirs to facilitate the learning of a behavioral policy through Reinforcement Learning (RL). Within an RL agent, a reservoir encodes the environment state before providing it to an action policy. We evaluate our approach on several 2D and 3D simulated environments. Our results show that the evolution of reservoirs can improve the learning of diverse challenging tasks. We study in particular three hypotheses: the use of an architecture combining reservoirs and reinforcement learning could enable (1) solving tasks with partial observability, (2) generating oscillatory dynamics that facilitate the learning of locomotion tasks, and (3) facilitating the generalization of learned behaviors to new tasks unknown during the evolution phase.