# Memory Capacity of Nonlinear Recurrent Networks: Is it Informative?

arXiv: https://arxiv.org/abs/2502.04832 · HTML: https://mlsanity.com/paper/2502.04832/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 31 references that are in the map.
- First posted on arXiv 7 Feb 2025.

## Builds on (most cited first, 15 of 15)

- [Poincar'e Embeddings for Learning Hierarchical Representations](https://mlsanity.com/paper/1705.08039/) (2017)
- [How to Construct Deep Recurrent Neural Networks](https://mlsanity.com/paper/1312.6026/) (2013)
- [On the Curse of Memory in Recurrent Neural Networks: Approximation and Optimization Analysis](https://mlsanity.com/paper/2009.07799/) (2020)
- [Memory and forecasting capacities of nonlinear recurrent networks](https://mlsanity.com/paper/2004.11234/) (2020)
- [A Structure-Preserving Kernel Method for Learning Hamiltonian Systems](https://mlsanity.com/paper/2403.10070/) (2024)
- [The Asymptotic Performance of Linear Echo State Neural Networks](https://mlsanity.com/paper/1603.07866/) (2016)
- [Memory of recurrent networks: Do we compute it right?](https://mlsanity.com/paper/2305.01457/) (2023)
- [Invariant kernels on Riemannian symmetric spaces: a harmonic-analytic approach](https://mlsanity.com/paper/2310.19270/) (2023)
- [The difference between memory and prediction in linear recurrent networks](https://mlsanity.com/paper/1706.09382/) (2017)
- [Short-term Memory of Deep RNN](https://mlsanity.com/paper/1802.00748/) (2018)
- [Echo State Networks with Self-Normalizing Activations on the Hyper-Sphere](https://mlsanity.com/paper/1903.11691/) (2019)
- [The Gaussian kernel on the circle and spaces that admit isometric embeddings of the circle](https://mlsanity.com/paper/2302.10623/) (2023)
- [Nonlinear memory capacity of parallel time-delay reservoir computers in the processing of multidimensional signals](https://mlsanity.com/paper/1510.03891/) (2015)
- [Geometric Learning with Positively Decomposable Kernels](https://mlsanity.com/paper/2310.13821/) (2023)
- [Understanding and Controlling Memory in Recurrent Neural Networks](https://mlsanity.com/paper/1902.07275/) (2019)

## Built on by (most cited first, 3 of 3)

- [Asymptotic evaluation of the information processing capacity in reservoir computing](https://mlsanity.com/paper/2502.15769/) (2025)
- [Information Processing Capacity of Stationary Physical Systems: Theory, Data-efficient Estimation Methods, and Photonic Demonstration](https://mlsanity.com/paper/2605.19152/) (2026)
- [Data-Specific Hyper-Parameter Design: A Paradigm Shift in Reservoir Computing](https://mlsanity.com/paper/2605.25221/) (2026)

## Abstract (arXiv)

The total memory capacity (MC) of linear recurrent neural networks (RNNs) has been proven to be equal to the rank of the corresponding Kalman controllability matrix, and it is almost surely maximal for connectivity and input weight matrices drawn from regular distributions. This fact questions the usefulness of this metric in distinguishing the performance of linear RNNs in the processing of stochastic signals. This work shows that the MC of random nonlinear RNNs yields arbitrary values within established upper and lower bounds depending exclusively on the scale of the input process. This confirms that the existing definition of MC in linear and nonlinear cases has no practical value.