mlsanityData to 14 Oct 2026

StableSSM: Alleviating the Curse of Memory in State-space Models through Stable Reparameterization

arXiv 2311.14495 ↗First posted 24 Nov 2023cs.LG cs.AI cs.CLcited by 30

Not on the map. Why?

Abstract (arXiv)

In this paper, we investigate the long-term memory learning capabilities of state-space models (SSMs) from the perspective of parameterization. We prove that state-space models without any reparameterization exhibit a memory limitation similar to that of traditional RNNs: the target relationships that can be stably approximated by state-space models must have an exponential decaying memory. Our analysis identifies this "curse of memory" as a result of the recurrent weights converging to a stability boundary, suggesting that a reparameterization technique can be effective. To this end, we introduce a class of reparameterization techniques for SSMs that effectively lift its memory limitations. Besides improving approximation capabilities, we further illustrate that a principled choice of reparameterization scheme can also enhance optimization stability. We validate our findings using synthetic datasets, language models and image classifications.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Builds on 24 of its 50 references that are in the map. First posted on arXiv 24 Nov 2023.

Builds on

24 of 50 references.

References · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2016Pointer Sentinel Mixture Modelscited by 3,096
  4. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  5. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  6. 2022Simplified State Space Layers for Sequence Modelingcited by 788
  7. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  8. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 613
  9. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 605
  10. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 499
  11. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 424
  12. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 408
12 more
  1. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 393
  2. 2022Long Range Language Modeling via Gated State Spacescited by 265
  3. 2023Small-scale proxies for large-scale Transformer training instabilitiescited by 183
  4. 2019Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networkscited by 162
  5. 2017Parallelizing Linear Recurrent Neural Nets Over Sequence Lengthcited by 127
  6. 2020Coupled Oscillatory Recurrent Neural Network (coRNN): An accurate and (gradient) stable architecture for learning long time dependenciescited by 82
  7. 2016Gradient Descent Learns Linear Dynamical Systemscited by 79
  8. 2023State-space Models with Layer-wise Nonlinearity are Universal Approximators with Exponential Decaying Memorycited by 39
  9. 2020On the Curse of Memory in Recurrent Neural Networks: Approximation and Optimization Analysiscited by 27
  10. 2023A Brief Survey on the Approximation Theory for Sequence Modellingcited by 15
  11. 2023Inverse Approximation Theory for Nonlinear Recurrent Neural Networkscited by 11
  12. 2023Improve Long-term Memory Learning Through Rescaling the Error Temporallycited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 12; 2025: 10; 2026: 8.

30 papers.

Sort:

By area:

  1. 2024ZigMa: A DiT-style Zigzag Mamba Diffusion Modelcited by 84
  2. 2024State Space Model for New-Generation Network Alternative to Transformers: A Surveycited by 59
  3. 2024The Expressive Capacity of State Space Models: A Formal Language Perspectivecited by 46
  4. 2024Visual Mamba: A Survey and New Outlookscited by 41
  5. 2024Tuning Frequency Bias of State Space Modelscited by 17
  6. 2024S7: Selective and Simplified State Space Layers for Sequence Modelingcited by 17
  7. 2024HOPE for a Robust Parameterization of Long-memory State Space Modelscited by 12
  8. 2024Time-SSM: Simplifying and Unifying State Space Models for Time Series Forecastingcited by 9
  9. 2025Advancing Intelligent Sequence Modeling: Evolution, Trade-offs, and Applications of State-Space Architectures from S4 to Mambacited by 9
  10. 2025Block-Biased Mamba for Long-Range Sequence Processingcited by 8
  11. 2024Towards a theory of learning dynamics in deep state space modelscited by 5
  12. 2025Generalization Error Analysis for Selective State-Space Models Through the Lens of Attentioncited by 5
18 more
  1. 2025Structured Kolmogorov-Arnold Neural ODEs for Interpretable Learning and Symbolic Discovery of Nonlinear Dynamicscited by 4
  2. 2024LongSSM: On the Length Extension of State-space Models in Language Modellingcited by 3
  3. 2024Revealing and Mitigating the Local Pattern Shortcuts of Mambacited by 3
  4. 2025A Deep State Space Model for Rainfall-Runoff Simulationscited by 3
  5. 2025CodeSSM: Towards State Space Models for Code Understandingcited by 3
  6. 2025CymbaDiff: Structured Spatial Diffusion for Sketch-based 3D Semantic Urban Scene Generationcited by 3
  7. 2024Length independent generalization bounds for deep SSM architectures via Rademacher contraction and stability constraintscited by 2
  8. 2025Numerical Investigation of Sequence Modeling Theory using Controllable Memory Functionscited by 1
  9. 2025LISM: Long-range Integrative State space Models via Input-Latent State Interactionscited by 1
  10. 2026Analysis of Long Range Dependency Understanding in State Space Modelscited by 1
  11. 2026Towards Understanding What State Space Models Learn About Codecited by 1
  12. 2026SHiPPO: Recurrent Memory with Transported Polynomial Projectionscited by 1
  13. 2025Performance-Efficiency Tradeoffs in Transformers: An Approximation Theory Perspectivecited by 0
  14. 2026Improved state mixing in higher-order and block diagonal linear recurrent networkscited by 0
  15. 2026Continuity Laws for Sequential Modelscited by 0
  16. 2026TIDES: Implicit Time-Awareness in Selective State Space Modelscited by 0
  17. 2026DSSMs: State Space Models with Explicit Memory via Delay Differential Equationscited by 0
  18. 2026The Cost of Long Memory: State, Context, and Stability Complexity in Sequence Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Memory Determines Learning Direction: A Theory of Gradient-Based Optimization in State Space Models
  2. 2024Recurrent neural networks: vanishing and exploding gradients are not the end of the story
  3. 2024Mathematical Formalism for Memory Compression in Selective State Space Models
  4. 2024Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothing
  5. 2026MaRK: Markov-adapted Recurrent Kernels for Dynamic Operator Conditioning in State Space Models
  6. 2025The Curious Case of In-Training Compression of State Space Models
  7. 2026Naju: A Native Discrete State-Space Model with Independent Retention and Writing for Long-Sequence Memory
  8. 2026On the Importance of Gating: Memorization vs. In-Context Learning in State Space Models