# Reservoir Computing for Fast, Simplified Reinforcement Learning on Memory Tasks

arXiv: https://arxiv.org/abs/2412.13093 · HTML: https://mlsanity.com/paper/2412.13093/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 3 of its 15 references that are in the map.
- First posted on arXiv 17 Dec 2024.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 3 of 3)

- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [POPGym: Benchmarking Partially Observable Reinforcement Learning](https://mlsanity.com/paper/2303.01859/) (2023)
- [Reservoirs learn to learn](https://mlsanity.com/paper/1909.07486/) (2019)

## Built on by (most cited first, 4 of 4)

- [Flexible Prefrontal Control over Hippocampal Episodic Memory for Goal-Directed Generalization](https://mlsanity.com/paper/2503.02303/) (2025)
- [A Method of Selective Attention for Reservoir Based Agents](https://mlsanity.com/paper/2502.21229/) (2025)
- [Meta-Learning to Explore via Memory Density Feedback](https://mlsanity.com/paper/2503.02831/) (2025)
- [Does Learning Recurrent Networks Matter in Partially Observable RL?](https://mlsanity.com/paper/2610.13197/) (2026)

## Abstract (arXiv)

Tasks in which rewards depend upon past information not available in the current observation set can only be solved by agents that are equipped with short-term memory. Usual choices for memory modules include trainable recurrent hidden layers, often with gated memory. Reservoir computing presents an alternative, in which a recurrent layer is not trained, but rather has a set of fixed, sparse recurrent weights. The weights are scaled to produce stable dynamical behavior such that the reservoir state contains a high-dimensional, nonlinear impulse response function of the inputs. An output decoder network can then be used to map the compressive history represented by the reservoir's state to any outputs, including agent actions or predictions. In this study, we find that reservoir computing greatly simplifies and speeds up reinforcement learning on memory tasks by (1) eliminating the need for backpropagation of gradients through time, (2) presenting all recent history simultaneously to the downstream network, and (3) performing many useful and generic nonlinear computations upstream from the trained modules. In particular, these findings offer significant benefit to meta-learning that depends primarily on efficient and highly general memory systems.