# Memory of recurrent networks: Do we compute it right?

arXiv: https://arxiv.org/abs/2305.01457 · HTML: https://mlsanity.com/paper/2305.01457/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 77 references that are in the map.
- First posted on arXiv 2 May 2023.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Echo state networks are universal](https://mlsanity.com/paper/1806.00797/) (2018)
- [Reservoir Computing Universality With Stochastic Inputs](https://mlsanity.com/paper/1807.02621/) (2018)
- [Fading memory echo state networks are universal](https://mlsanity.com/paper/2010.12047/) (2020)
- [On the Curse of Memory in Recurrent Neural Networks: Approximation and Optimization Analysis](https://mlsanity.com/paper/2009.07799/) (2020)
- [Designing Universal Causal Deep Learning Models: The Geometric (Hyper)Transformer](https://mlsanity.com/paper/2201.13094/) (2022)
- [Designing Universal Causal Deep Learning Models: The Case of Infinite-Dimensional Dynamical Systems from Stochastic Analysis](https://mlsanity.com/paper/2210.13300/) (2022)
- [Learning strange attractors with reservoir systems](https://mlsanity.com/paper/2108.05024/) (2021)
- [Determination of the edge of criticality in echo state networks through Fisher information maximization](https://mlsanity.com/paper/1603.03685/) (2016)
- [Memory and forecasting capacities of nonlinear recurrent networks](https://mlsanity.com/paper/2004.11234/) (2020)
- [Dimension reduction in recurrent networks by canonicalization](https://mlsanity.com/paper/2007.12141/) (2020)
- [Krylov Subspace Method for Nonlinear Dynamical Systems with Random Noise](https://mlsanity.com/paper/1909.03634/) (2019)
- [The Asymptotic Performance of Linear Echo State Neural Networks](https://mlsanity.com/paper/1603.07866/) (2016)
- [Sparsity in Reservoir Computing Neural Networks](https://mlsanity.com/paper/2006.02957/) (2020)
- [The difference between memory and prediction in linear recurrent networks](https://mlsanity.com/paper/1706.09382/) (2017)
- [Short-term Memory of Deep RNN](https://mlsanity.com/paper/1802.00748/) (2018)
- [Echo State Networks with Self-Normalizing Activations on the Hyper-Sphere](https://mlsanity.com/paper/1903.11691/) (2019)
- [Transport in reservoir computing](https://mlsanity.com/paper/2209.07946/) (2022)
- [Nonlinear memory capacity of parallel time-delay reservoir computers in the processing of multidimensional signals](https://mlsanity.com/paper/1510.03891/) (2015)
- [Memorization Capacity of Neural Networks with Conditional Computation](https://mlsanity.com/paper/2303.11247/) (2023)
- [Memory and Information Processing in Recurrent Neural Networks](https://mlsanity.com/paper/1604.06929/) (2016)
- [Distributed Sequence Memory of Multidimensional Inputs in Recurrent Networks](https://mlsanity.com/paper/1605.08346/) (2016)
- [Understanding and Controlling Memory in Recurrent Neural Networks](https://mlsanity.com/paper/1902.07275/) (2019)
- [Tailoring Artificial Neural Networks for Optimal Learning](https://mlsanity.com/paper/1707.02469/) (2017)

## Built on by (most cited first, 6 of 6)

- [Memory Capacity of Nonlinear Recurrent Networks: Is it Informative?](https://mlsanity.com/paper/2502.04832/) (2025)
- [How noise affects memory in linear recurrent networks](https://mlsanity.com/paper/2409.03187/) (2024)
- [Memory Determines Learning Direction: A Theory of Gradient-Based Optimization in State Space Models](https://mlsanity.com/paper/2510.00563/) (2025)
- [Information Processing Capacity of Stationary Physical Systems: Theory, Data-efficient Estimation Methods, and Photonic Demonstration](https://mlsanity.com/paper/2605.19152/) (2026)
- [Data-Specific Hyper-Parameter Design: A Paradigm Shift in Reservoir Computing](https://mlsanity.com/paper/2605.25221/) (2026)
- [Empirical Boundaries of Reservoir Computing for Few-Shot Sequence Pattern Tasks](https://mlsanity.com/paper/2610.19997/) (2026)

## Abstract (arXiv)

Numerical evaluations of the memory capacity (MC) of recurrent neural networks reported in the literature often contradict well-established theoretical bounds. In this paper, we study the case of linear echo state networks, for which the total memory capacity has been proven to be equal to the rank of the corresponding Kalman controllability matrix. We shed light on various reasons for the inaccurate numerical estimations of the memory, and we show that these issues, often overlooked in the recent literature, are of an exclusively numerical nature. More explicitly, we prove that when the Krylov structure of the linear MC is ignored, a gap between the theoretical MC and its empirical counterpart is introduced. As a solution, we develop robust numerical approaches by exploiting a result of MC neutrality with respect to the input mask matrix. Simulations show that the memory curves that are recovered using the proposed methods fully agree with the theory.