# Memory and forecasting capacities of nonlinear recurrent networks

arXiv: https://arxiv.org/abs/2004.11234 · HTML: https://mlsanity.com/paper/2004.11234/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Apr 2020.

## Built on by (most cited first, 11 of 11)

- [Learning strange attractors with reservoir systems](https://mlsanity.com/paper/2108.05024/) (2021)
- [Infinite-dimensional reservoir computing](https://mlsanity.com/paper/2304.00490/) (2023)
- [Memory of recurrent networks: Do we compute it right?](https://mlsanity.com/paper/2305.01457/) (2023)
- [Memory Capacity of Nonlinear Recurrent Networks: Is it Informative?](https://mlsanity.com/paper/2502.04832/) (2025)
- [How noise affects memory in linear recurrent networks](https://mlsanity.com/paper/2409.03187/) (2024)
- [Interpretable Design of Reservoir Computing Networks using Realization Theory](https://mlsanity.com/paper/2112.06891/) (2021)
- [Empirical Analysis of Limits for Memory Distance in Recurrent Neural Networks](https://mlsanity.com/paper/2212.11085/) (2022)
- [Separation capacity of linear reservoirs with random connectivity matrix](https://mlsanity.com/paper/2404.17429/) (2024)
- [Memory Determines Learning Direction: A Theory of Gradient-Based Optimization in State Space Models](https://mlsanity.com/paper/2510.00563/) (2025)
- [Information Processing Capacity of Stationary Physical Systems: Theory, Data-efficient Estimation Methods, and Photonic Demonstration](https://mlsanity.com/paper/2605.19152/) (2026)
- [Data-Specific Hyper-Parameter Design: A Paradigm Shift in Reservoir Computing](https://mlsanity.com/paper/2605.25221/) (2026)

## Abstract (arXiv)

The notion of memory capacity, originally introduced for echo state and linear networks with independent inputs, is generalized to nonlinear recurrent networks with stationary but dependent inputs. The presence of dependence in the inputs makes natural the introduction of the network forecasting capacity, that measures the possibility of forecasting time series values using network states. Generic bounds for memory and forecasting capacities are formulated in terms of the number of neurons of the nonlinear recurrent network and the autocovariance function or the spectral density of the input. These bounds generalize well-known estimates in the literature to a dependent inputs setup. Finally, for the particular case of linear recurrent networks with independent inputs it is proved that the memory capacity is given by the rank of the associated controllability matrix, a fact that has been for a long time assumed to be true without proof by the community.