mlsanityData to 14 Oct 2026

Inverse Approximation Theory for Nonlinear Recurrent Neural Networks

arXiv 2305.19190 ↗First posted 30 May 2023cs.LG cs.AI math.DScited by 11

Not on the map. Why?

Abstract (arXiv)

We prove an inverse approximation theorem for the approximation of nonlinear sequence-to-sequence relationships using recurrent neural networks (RNNs). This is a so-called Bernstein-type result in approximation theory, which deduces properties of a target function under the assumption that it can be effectively approximated by a hypothesis space. In particular, we show that nonlinear sequence relationships that can be stably approximated by nonlinear RNNs must have an exponential decaying memory structure - a notion that can be made precise. This extends the previously identified curse of memory in linear RNNs into the general nonlinear setting, and quantifies the essential limitations of the RNN architecture for learning sequential relationships with long-term memory. Based on the analysis, we propose a principled reparameterization method to overcome the limitations. Our theoretical results are confirmed by numerical experiments. The code has been released in https://github.com/radarFudan/Curse-of-memory

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 12 of its 37 references that are in the map. First posted on arXiv 30 May 2023.

Builds on

12 of 37 references.

References · 12

  1. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  2. 2022Simplified State Space Layers for Sequence Modelingcited by 788
  3. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 605
  4. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 408
  5. 2019Cheap Orthogonal Constraints in Neural Networks: A Simple Parametrization of the Orthogonal and Unitary Groupcited by 134
  6. 2020Learning Long-Term Dependencies in Irregularly-Sampled Time Seriescited by 83
  7. 2018Reservoir Computing Universality With Stochastic Inputscited by 48
  8. 2020Fading memory echo state networks are universalcited by 32
  9. 2020On the Curse of Memory in Recurrent Neural Networks: Approximation and Optimization Analysiscited by 27
  10. 2023A Brief Survey on the Approximation Theory for Sequence Modellingcited by 15
  11. 2020Learning Recurrent Neural Net Models of Nonlinear Systemscited by 11
  12. 2021Approximation Theory of Convolutional Architectures for Time Series Modellingcited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 3; 2025: 3; 2026: 1.

11 papers.

  1. 2023State-space Models with Layer-wise Nonlinearity are Universal Approximators with Exponential Decaying Memorycited by 39
  2. 2023StableSSM: Alleviating the Curse of Memory in State-space Models through Stable Reparameterizationcited by 30
  3. 2024Recurrent neural networks: vanishing and exploding gradients are not the end of the storycited by 25
  4. 2024From Generalization Analysis to Optimization Designs for State Space Modelscited by 11
  5. 2025Understanding Input Selectivity in Mamba: Impact on Approximation Power, Memorization, and Associative Recall Capacitycited by 7
  6. 2024LongSSM: On the Length Extension of State-space Models in Language Modellingcited by 3
  7. 2023Improve Long-term Memory Learning Through Rescaling the Error Temporallycited by 1
  8. 2025Numerical Investigation of Sequence Modeling Theory using Controllable Memory Functionscited by 1
  9. 2023HyperSNN: A new efficient and robust deep learning model for resource constrained control applicationscited by 0
  10. 2025Performance-Efficiency Tradeoffs in Transformers: An Approximation Theory Perspectivecited by 0
  11. 2026The Cost of Long Memory: State, Context, and Stability Complexity in Sequence Modelscited by 0