mlsanityData to 7 Oct 2026

Memory Capacity of Nonlinear Recurrent Networks: Is it Informative?

arXiv 2502.04832 ↗First posted 7 Feb 2025cs.LG stat.MLcited by 3

Not on the map. Why?

Abstract (arXiv)

The total memory capacity (MC) of linear recurrent neural networks (RNNs) has been proven to be equal to the rank of the corresponding Kalman controllability matrix, and it is almost surely maximal for connectivity and input weight matrices drawn from regular distributions. This fact questions the usefulness of this metric in distinguishing the performance of linear RNNs in the processing of stochastic signals. This work shows that the MC of random nonlinear RNNs yields arbitrary values within established upper and lower bounds depending exclusively on the scale of the input process. This confirms that the existing definition of MC in linear and nonlinear cases has no practical value.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 15 of its 31 references that are in the map. First posted on arXiv 7 Feb 2025.

Builds on

15 of 31 references.

References · 15

  1. 2017Poincar'e Embeddings for Learning Hierarchical Representationscited by 674
  2. 2013How to Construct Deep Recurrent Neural Networkscited by 64
  3. 2020On the Curse of Memory in Recurrent Neural Networks: Approximation and Optimization Analysiscited by 26
  4. 2020Memory and forecasting capacities of nonlinear recurrent networkscited by 11
  5. 2024A Structure-Preserving Kernel Method for Learning Hamiltonian Systemscited by 9
  6. 2016The Asymptotic Performance of Linear Echo State Neural Networkscited by 6
  7. 2023Invariant kernels on Riemannian symmetric spaces: a harmonic-analytic approachcited by 6
  8. 2017The difference between memory and prediction in linear recurrent networkscited by 5
  9. 2018Short-term Memory of Deep RNNcited by 5
  10. 2019Echo State Networks with Self-Normalizing Activations on the Hyper-Spherecited by 5
  11. 2023The Gaussian kernel on the circle and spaces that admit isometric embeddings of the circlecited by 5
  12. 2023Memory of recurrent networks: Do we compute it right?cited by 5
3 more
  1. 2015Nonlinear memory capacity of parallel time-delay reservoir computers in the processing of multidimensional signalscited by 3
  2. 2023Geometric Learning with Positively Decomposable Kernelscited by 3
  3. 2019Understanding and Controlling Memory in Recurrent Neural Networkscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Asymptotic evaluation of the information processing capacity in reservoir computingcited by 3
  2. 2026Information Processing Capacity of Stationary Physical Systems: Theory, Data-efficient Estimation Methods, and Photonic Demonstrationcited by 0
  3. 2026Data-Specific Hyper-Parameter Design: A Paradigm Shift in Reservoir Computingcited by 0