# On the Curse of Memory in Recurrent Neural Networks: Approximation and Optimization Analysis

arXiv: https://arxiv.org/abs/2009.07799 · HTML: https://mlsanity.com/paper/2009.07799/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Sep 2020.

## Built on by (most cited first, 27 of 27)

- [On the Generalization Properties of Diffusion Models](https://mlsanity.com/paper/2311.01797/) (2023)
- [RNNs are not Transformers (Yet): The Key Bottleneck on In-context Retrieval](https://mlsanity.com/paper/2402.18510/) (2024)
- [A Framework for Machine Learning of Model Error in Dynamical Systems](https://mlsanity.com/paper/2107.06658/) (2021)
- [StableSSM: Alleviating the Curse of Memory in State-space Models through Stable Reparameterization](https://mlsanity.com/paper/2311.14495/) (2023)
- [Approximation Rate of the Transformer Architecture for Sequence Modeling](https://mlsanity.com/paper/2305.18475/) (2023)
- [A Brief Survey on the Approximation Theory for Sequence Modelling](https://mlsanity.com/paper/2302.13752/) (2023)
- [The Discovery of Dynamics via Linear Multistep Methods and Deep Learning: Error Estimation](https://mlsanity.com/paper/2103.11488/) (2021)
- [Inverse Approximation Theory for Nonlinear Recurrent Neural Networks](https://mlsanity.com/paper/2305.19190/) (2023)
- [From Generalization Analysis to Optimization Designs for State Space Models](https://mlsanity.com/paper/2405.02670/) (2024)
- [Approximation Theory of Convolutional Architectures for Time Series Modelling](https://mlsanity.com/paper/2107.09355/) (2021)
- [Deep Neural Network Approximation of Invariant Functions through Dynamical Systems](https://mlsanity.com/paper/2208.08707/) (2022)
- [Koopman-informed recurrent neural networks](https://mlsanity.com/paper/2410.23467/) (2024)
- [Calibrating multi-dimensional complex ODE from noisy data via deep neural networks](https://mlsanity.com/paper/2106.03591/) (2021)
- [Memory of recurrent networks: Do we compute it right?](https://mlsanity.com/paper/2305.01457/) (2023)
- [Metric Entropy Limits on Recurrent Neural Network Learning of Linear Dynamical Systems](https://mlsanity.com/paper/2105.02556/) (2021)
- [LongSSM: On the Length Extension of State-space Models in Language Modelling](https://mlsanity.com/paper/2406.02080/) (2024)
- [Memory Capacity of Nonlinear Recurrent Networks: Is it Informative?](https://mlsanity.com/paper/2502.04832/) (2025)
- [Accuracy, Memory Efficiency and Generalization: A Comparative Study on Liquid Neural Networks and Recurrent Neural Networks](https://mlsanity.com/paper/2510.07578/) (2025)
- [Fading memory and the convolution theorem](https://mlsanity.com/paper/2408.07386/) (2024)
- [Path classification by stochastic linear recurrent neural networks](https://mlsanity.com/paper/2108.03090/) (2021)
- [Improve Long-term Memory Learning Through Rescaling the Error Temporally](https://mlsanity.com/paper/2307.11462/) (2023)
- [Separation capacity of linear reservoirs with random connectivity matrix](https://mlsanity.com/paper/2404.17429/) (2024)
- [A finite-sample generalization bound for stable LPV systems](https://mlsanity.com/paper/2405.10054/) (2024)
- [Performance-Efficiency Tradeoffs in Transformers: An Approximation Theory Perspective](https://mlsanity.com/paper/2510.03784/) (2025)
- [A Taxonomy of Cognitive Capability Gaps in Generative and Agentic AI](https://mlsanity.com/paper/2608.02553/) (2026)
- [Finite-Horizon Fisher Memory in Two-Sided Power-Bounded Recurrent Systems](https://mlsanity.com/paper/2609.39800/) (2026)
- [The Cost of Long Memory: State, Context, and Stability Complexity in Sequence Models](https://mlsanity.com/paper/2610.08816/) (2026)

## Abstract (arXiv)

We study the approximation properties and optimization dynamics of recurrent neural networks (RNNs) when applied to learn input-output relationships in temporal data. We consider the simple but representative setting of using continuous-time linear RNNs to learn from data generated by linear relationships. Mathematically, the latter can be understood as a sequence of linear functionals. We prove a universal approximation theorem of such linear functionals, and characterize the approximation rate and its relation with memory. Moreover, we perform a fine-grained dynamical analysis of training linear RNNs, which further reveal the intricate interactions between memory and learning. A unifying theme uncovered is the non-trivial effect of memory, a notion that can be made precise in our framework, on approximation and optimization: when there is long term memory in the target, it takes a large number of neurons to approximate it. Moreover, the training process will suffer from slow downs. In particular, both of these effects become exponentially more pronounced with memory - a phenomenon we call the "curse of memory". These analyses represent a basic step towards a concrete mathematical understanding of new phenomenon that may arise in learning temporal relationships using recurrent architectures.