# Deep Q-network using reservoir computing with multi-layered readout

arXiv: https://arxiv.org/abs/2203.01465 · HTML: https://mlsanity.com/paper/2203.01465/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 41 references that are in the map.
- First posted on arXiv 3 Mar 2022.

## Builds on (most cited first, 13 of 13)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Deep Reinforcement Learning with Double Q-learning](https://mlsanity.com/paper/1509.06461/) (2015)
- [Deep Recurrent Q-Learning for Partially Observable MDPs](https://mlsanity.com/paper/1507.06527/) (2015)
- [Distributed Prioritized Experience Replay](https://mlsanity.com/paper/1803.00933/) (2018)
- [Recurrent Neural Network Regularization](https://mlsanity.com/paper/1409.2329/) (2014)
- [Playing FPS Games with Deep Reinforcement Learning](https://mlsanity.com/paper/1609.05521/) (2016)
- [Reinforcement Learning with Convolutional Reservoir Computing](https://mlsanity.com/paper/1912.04161/) (2019)
- [Model-Size Reduction for Reservoir Computing by Concatenating Internal States Through Time](https://mlsanity.com/paper/2006.06218/) (2020)
- [Deep Echo State Q-Network (DEQN) and Its Application in Dynamic Spectrum Sharing for 5G and Beyond](https://mlsanity.com/paper/2010.05449/) (2020)
- [Performing Deep Recurrent Double Q-Learning for Atari Games](https://mlsanity.com/paper/1908.06040/) (2019)
- [Deep-ESN: A Multiple Projection-encoding Hierarchical Reservoir Computing Framework](https://mlsanity.com/paper/1711.05255/) (2017)

## Built on by (most cited first, 2 of 2)

- [Chaos-based reinforcement learning with TD3](https://mlsanity.com/paper/2405.09086/) (2024)
- [Modulating Reservoir Dynamics via Reinforcement Learning for Efficient Robot Skill Synthesis](https://mlsanity.com/paper/2411.10991/) (2024)

## Abstract (arXiv)

Recurrent neural network (RNN) based reinforcement learning (RL) is used for learning context-dependent tasks and has also attracted attention as a method with remarkable learning performance in recent research. However, RNN-based RL has some issues that the learning procedures tend to be more computationally expensive, and training with backpropagation through time (BPTT) is unstable because of vanishing/exploding gradients problem. An approach with replay memory introducing reservoir computing has been proposed, which trains an agent without BPTT and avoids these issues. The basic idea of this approach is that observations from the environment are input to the reservoir network, and both the observation and the reservoir output are stored in the memory. This paper shows that the performance of this method improves by using a multi-layered neural network for the readout layer, which regularly consists of a single linear layer. The experimental results show that using multi-layered readout improves the learning performance of four classical control tasks that require time-series processing.