# Short-term Memory of Deep RNN

arXiv: https://arxiv.org/abs/1802.00748 · HTML: https://mlsanity.com/paper/1802.00748/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Feb 2018.

## Built on by (most cited first, 5 of 5)

- [Memory of recurrent networks: Do we compute it right?](https://mlsanity.com/paper/2305.01457/) (2023)
- [Memory Capacity of Nonlinear Recurrent Networks: Is it Informative?](https://mlsanity.com/paper/2502.04832/) (2025)
- [Including STDP to eligibility propagation in multi-layer recurrent spiking neural networks](https://mlsanity.com/paper/2201.07602/) (2022)
- [Improve Long-term Memory Learning Through Rescaling the Error Temporally](https://mlsanity.com/paper/2307.11462/) (2023)
- [Information Processing Capacity of Stationary Physical Systems: Theory, Data-efficient Estimation Methods, and Photonic Demonstration](https://mlsanity.com/paper/2605.19152/) (2026)

## Abstract (arXiv)

The extension of deep learning towards temporal data processing is gaining an increasing research interest. In this paper we investigate the properties of state dynamics developed in successive levels of deep recurrent neural networks (RNNs) in terms of short-term memory abilities. Our results reveal interesting insights that shed light on the nature of layering as a factor of RNN design. Noticeably, higher layers in a hierarchically organized RNN architecture results to be inherently biased towards longer memory spans even prior to training of the recurrent connections. Moreover, in the context of Reservoir Computing framework, our analysis also points out the benefit of a layered recurrent organization as an efficient approach to improve the memory skills of reservoir models.