# StableSSM: Alleviating the Curse of Memory in State-space Models through Stable Reparameterization

arXiv: https://arxiv.org/abs/2311.14495 · HTML: https://mlsanity.com/paper/2311.14495/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 50 references that are in the map.
- First posted on arXiv 24 Nov 2023.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [RWKV: Reinventing RNNs for the Transformer Era](https://mlsanity.com/paper/2305.13048/) (2023)
- [Simplified State Space Layers for Sequence Modeling](https://mlsanity.com/paper/2208.04933/) (2022)
- [Long Range Arena: A Benchmark for Efficient Transformers](https://mlsanity.com/paper/2011.04006/) (2020)
- [Retentive Network: A Successor to Transformer for Large Language Models](https://mlsanity.com/paper/2307.08621/) (2023)
- [HiPPO: Recurrent Memory with Optimal Polynomial Projections](https://mlsanity.com/paper/2008.07669/) (2020)
- [Hungry Hungry Hippos: Towards Language Modeling with State Space Models](https://mlsanity.com/paper/2212.14052/) (2022)
- [On the Parameterization and Initialization of Diagonal State Space Models](https://mlsanity.com/paper/2206.11893/) (2022)
- [Resurrecting Recurrent Neural Networks for Long Sequences](https://mlsanity.com/paper/2303.06349/) (2023)
- [Hyena Hierarchy: Towards Larger Convolutional Language Models](https://mlsanity.com/paper/2302.10866/) (2023)
- [Long Range Language Modeling via Gated State Spaces](https://mlsanity.com/paper/2206.13947/) (2022)
- [Small-scale proxies for large-scale Transformer training instabilities](https://mlsanity.com/paper/2309.14322/) (2023)
- [Towards Explaining the Regularization Effect of Initial Large Learning Rate in Training Neural Networks](https://mlsanity.com/paper/1907.04595/) (2019)
- [Parallelizing Linear Recurrent Neural Nets Over Sequence Length](https://mlsanity.com/paper/1709.04057/) (2017)
- [Coupled Oscillatory Recurrent Neural Network (coRNN): An accurate and (gradient) stable architecture for learning long time dependencies](https://mlsanity.com/paper/2010.00951/) (2020)
- [Gradient Descent Learns Linear Dynamical Systems](https://mlsanity.com/paper/1609.05191/) (2016)
- [State-space Models with Layer-wise Nonlinearity are Universal Approximators with Exponential Decaying Memory](https://mlsanity.com/paper/2309.13414/) (2023)
- [On the Curse of Memory in Recurrent Neural Networks: Approximation and Optimization Analysis](https://mlsanity.com/paper/2009.07799/) (2020)
- [A Brief Survey on the Approximation Theory for Sequence Modelling](https://mlsanity.com/paper/2302.13752/) (2023)
- [Inverse Approximation Theory for Nonlinear Recurrent Neural Networks](https://mlsanity.com/paper/2305.19190/) (2023)
- [Improve Long-term Memory Learning Through Rescaling the Error Temporally](https://mlsanity.com/paper/2307.11462/) (2023)

## Built on by (most cited first, 30 of 30)

- [ZigMa: A DiT-style Zigzag Mamba Diffusion Model](https://mlsanity.com/paper/2403.13802/) (2024)
- [State Space Model for New-Generation Network Alternative to Transformers: A Survey](https://mlsanity.com/paper/2404.09516/) (2024)
- [The Expressive Capacity of State Space Models: A Formal Language Perspective](https://mlsanity.com/paper/2405.17394/) (2024)
- [Visual Mamba: A Survey and New Outlooks](https://mlsanity.com/paper/2404.18861/) (2024)
- [Tuning Frequency Bias of State Space Models](https://mlsanity.com/paper/2410.02035/) (2024)
- [S7: Selective and Simplified State Space Layers for Sequence Modeling](https://mlsanity.com/paper/2410.03464/) (2024)
- [HOPE for a Robust Parameterization of Long-memory State Space Models](https://mlsanity.com/paper/2405.13975/) (2024)
- [Time-SSM: Simplifying and Unifying State Space Models for Time Series Forecasting](https://mlsanity.com/paper/2405.16312/) (2024)
- [Advancing Intelligent Sequence Modeling: Evolution, Trade-offs, and Applications of State-Space Architectures from S4 to Mamba](https://mlsanity.com/paper/2503.18970/) (2025)
- [Block-Biased Mamba for Long-Range Sequence Processing](https://mlsanity.com/paper/2505.09022/) (2025)
- [Towards a theory of learning dynamics in deep state space models](https://mlsanity.com/paper/2407.07279/) (2024)
- [Generalization Error Analysis for Selective State-Space Models Through the Lens of Attention](https://mlsanity.com/paper/2502.01473/) (2025)
- [Structured Kolmogorov-Arnold Neural ODEs for Interpretable Learning and Symbolic Discovery of Nonlinear Dynamics](https://mlsanity.com/paper/2506.18339/) (2025)
- [LongSSM: On the Length Extension of State-space Models in Language Modelling](https://mlsanity.com/paper/2406.02080/) (2024)
- [Revealing and Mitigating the Local Pattern Shortcuts of Mamba](https://mlsanity.com/paper/2410.15678/) (2024)
- [A Deep State Space Model for Rainfall-Runoff Simulations](https://mlsanity.com/paper/2501.14980/) (2025)
- [CodeSSM: Towards State Space Models for Code Understanding](https://mlsanity.com/paper/2505.01475/) (2025)
- [CymbaDiff: Structured Spatial Diffusion for Sketch-based 3D Semantic Urban Scene Generation](https://mlsanity.com/paper/2510.13245/) (2025)
- [Length independent generalization bounds for deep SSM architectures via Rademacher contraction and stability constraints](https://mlsanity.com/paper/2405.20278/) (2024)
- [Numerical Investigation of Sequence Modeling Theory using Controllable Memory Functions](https://mlsanity.com/paper/2506.05678/) (2025)
- [LISM: Long-range Integrative State space Models via Input-Latent State Interactions](https://mlsanity.com/paper/2509.04226/) (2025)
- [Analysis of Long Range Dependency Understanding in State Space Models](https://mlsanity.com/paper/2601.13048/) (2026)
- [Towards Understanding What State Space Models Learn About Code](https://mlsanity.com/paper/2602.06774/) (2026)
- [SHiPPO: Recurrent Memory with Transported Polynomial Projections](https://mlsanity.com/paper/2607.03055/) (2026)
- [Performance-Efficiency Tradeoffs in Transformers: An Approximation Theory Perspective](https://mlsanity.com/paper/2510.03784/) (2025)
- [Improved state mixing in higher-order and block diagonal linear recurrent networks](https://mlsanity.com/paper/2602.12021/) (2026)
- [Continuity Laws for Sequential Models](https://mlsanity.com/paper/2605.08539/) (2026)
- [TIDES: Implicit Time-Awareness in Selective State Space Models](https://mlsanity.com/paper/2605.09742/) (2026)
- [DSSMs: State Space Models with Explicit Memory via Delay Differential Equations](https://mlsanity.com/paper/2607.10244/) (2026)
- [The Cost of Long Memory: State, Context, and Stability Complexity in Sequence Models](https://mlsanity.com/paper/2610.08816/) (2026)

## Abstract (arXiv)

In this paper, we investigate the long-term memory learning capabilities of state-space models (SSMs) from the perspective of parameterization. We prove that state-space models without any reparameterization exhibit a memory limitation similar to that of traditional RNNs: the target relationships that can be stably approximated by state-space models must have an exponential decaying memory. Our analysis identifies this "curse of memory" as a result of the recurrent weights converging to a stability boundary, suggesting that a reparameterization technique can be effective. To this end, we introduce a class of reparameterization techniques for SSMs that effectively lift its memory limitations. Besides improving approximation capabilities, we further illustrate that a principled choice of reparameterization scheme can also enhance optimization stability. We validate our findings using synthetic datasets, language models and image classifications.