mlsanityData to 14 Oct 2026

State-space Models with Layer-wise Nonlinearity are Universal Approximators with Exponential Decaying Memory

arXiv 2309.13414 ↗First posted 23 Sep 2023cs.LG cs.AI math.DScited by 39

Not on the map. Why?

Abstract (arXiv)

State-space models have gained popularity in sequence modelling due to their simple and efficient network structures. However, the absence of nonlinear activation along the temporal direction limits the model's capacity. In this paper, we prove that stacking state-space models with layer-wise nonlinear activation is sufficient to approximate any continuous sequence-to-sequence relationship. Our findings demonstrate that the addition of layer-wise nonlinear activation enhances the model's capacity to learn complex sequence patterns. Meanwhile, it can be seen both theoretically and empirically that the state-space models do not fundamentally resolve the issue of exponential decaying memory. Theoretical results are justified by numerical verifications.

Key facts. Cited by 39 papers in the map (data through 14 Oct 2026). Builds on 17 of its 29 references that are in the map. First posted on arXiv 23 Sep 2023.

Builds on

17 of 29 references.

References · 17

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modelingcited by 1,476
  3. 2022Simplified State Space Layers for Sequence Modelingcited by 788
  4. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  5. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 605
  6. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 499
  7. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 424
  8. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 408
  9. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 393
  10. 2015The Power of Depth for Feedforward Neural Networkscited by 183
  11. 2017Parallelizing Linear Recurrent Neural Nets Over Sequence Lengthcited by 127
  12. 2022How to Train Your HiPPO: State Space Models with Generalized Orthogonal Basis Projectionscited by 122
5 more
  1. 2021CKConv: Continuous Kernel Convolution For Sequential Datacited by 120
  2. 2022Pretraining Without Attentioncited by 51
  3. 2023A Brief Survey on the Approximation Theory for Sequence Modellingcited by 15
  4. 2020Learning Recurrent Neural Net Models of Nonlinear Systemscited by 11
  5. 2023Inverse Approximation Theory for Nonlinear Recurrent Neural Networkscited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 20; 2025: 9; 2026: 9.

39 papers.

Sort:

By area:

  1. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  2. 2024Graph Mamba: Towards Learning on Graphs with State Space Modelscited by 86
  3. 2024ZigMa: A DiT-style Zigzag Mamba Diffusion Modelcited by 84
  4. 2024Theoretical Foundations of Deep Selective State-Space Modelscited by 74
  5. 2024State Space Model for New-Generation Network Alternative to Transformers: A Surveycited by 59
  6. 2024The Expressive Capacity of State Space Models: A Formal Language Perspectivecited by 46
  7. 2023StableSSM: Alleviating the Curse of Memory in State-space Models through Stable Reparameterizationcited by 30
  8. 2024Understanding the differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networkscited by 26
  9. 2024Learning Long Range Dependencies on Graphs via Random Walkscited by 18
  10. 2024Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothingcited by 18
  11. 2024Tuning Frequency Bias of State Space Modelscited by 17
  12. 2024HOPE for a Robust Parameterization of Long-memory State Space Modelscited by 12
27 more
  1. 2024Analog In-Memory Computing Attention Mechanism for Fast and Energy-Efficient Large Language Modelscited by 11
  2. 2025From Markov to Laplace: How Mamba In-Context Learns Markov Chainscited by 11
  3. 2025Block-Biased Mamba for Long-Range Sequence Processingcited by 8
  4. 2024Mamba-FSCIL: Dynamic Adaptation with Selective State Space Model for Few-Shot Class-Incremental Learningcited by 7
  5. 2024State Space Models are Provably Comparable to Transformers in Dynamic Token Selectioncited by 6
  6. 2024Exploring Adversarial Robustness of Deep State Space Modelscited by 4
  7. 2024LongSSM: On the Length Extension of State-space Models in Language Modellingcited by 3
  8. 2024On the low-shot transferability of [V]-Mambacited by 1
  9. 2025Numerical Investigation of Sequence Modeling Theory using Controllable Memory Functionscited by 1
  10. 2025The Effect of Depth on the Expressivity of Deep Linear State-Space Modelscited by 1
  11. 2025Upper Approximation Bounds for Neural Oscillatorscited by 1
  12. 2026SHiPPO: Recurrent Memory with Transported Polynomial Projectionscited by 1
  13. 2024A finite-sample generalization bound for stable LPV systemscited by 0
  14. 2024Universal In-Context Approximation By Prompting Fully Recurrent Modelscited by 0
  15. 2024GLAudio Listens to the Sound of the Graphcited by 0
  16. 2025Revisiting Reset Mechanisms in Spiking Neural Networks for Sequential Modeling: Specialized Discretization for Binary Activated RNNcited by 0
  17. 2025FaRMamba: Frequency-based learning and Reconstruction aided Mamba for Medical Segmentationcited by 0
  18. 2025Systolic Array-based Accelerator for Structured State-Space Modelscited by 0
  19. 2025MambaX: Image Super-Resolution with State Predictive Controlcited by 0
  20. 2026Extending SSMs with the Exponentially Weighted Signaturecited by 0
  21. 2026On the Role of Depth in the Expressivity of RNNscited by 0
  22. 2026An explicit operator explains end-to-end computation in the modern neural networks used for sequence and language modelingcited by 0
  23. 2026Universal Time Series Generation with Neural Controlled Differential Equationscited by 0
  24. 2026Neural Events: Discrete Asynchronous Autoencoders for Event-Based Visioncited by 0
  25. 2026DSSMs: State Space Models with Explicit Memory via Delay Differential Equationscited by 0
  26. 2026Fractional State Space Transition for Long Sequence Modelingcited by 0
  27. 2026The Cost of Long Memory: State, Context, and Stability Complexity in Sequence Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Mathematical Formalism for Memory Compression in Selective State Space Models
  2. 2025Advancing Intelligent Sequence Modeling: Evolution, Trade-offs, and Applications of State-Space Architectures from S4 to Mamba
  3. 2021Efficiently Modeling Long Sequences with Structured State Spaces
  4. 2024S7: Selective and Simplified State Space Layers for Sequence Modeling
  5. 2024HiPPO-Prophecy: State-Space Models can Provably Learn Dynamical Systems in Context
  6. 2026MuonSSM: Orthogonalizing State Space Models for Sequence Modeling
  7. 2025Memory Determines Learning Direction: A Theory of Gradient-Based Optimization in State Space Models
  8. 2025Learning to Dissipate Energy in Oscillatory State-Space Models