mlsanityData to 7 Oct 2026

Generalization Error Analysis for Selective State-Space Models Through the Lens of Attention

arXiv 2502.01473 ↗First posted 3 Feb 2025cs.LGcited by 5

On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language

Abstract (arXiv)

State-space models (SSMs) have recently emerged as a compelling alternative to Transformers for sequence modeling tasks. This paper presents a theoretical generalization analysis of selective SSMs, the core architectural component behind the Mamba model. We derive a novel covering number-based generalization bound for selective SSMs, building upon recent theoretical advances in the analysis of Transformer models. Using this result, we analyze how the spectral abscissa of the continuous-time state matrix influences the model's stability during training and its ability to generalize across sequence lengths. We empirically validate our findings on a synthetic majority task, the IMDb sentiment classification benchmark, and the ListOps task, demonstrating how our theoretical insights translate into practical model behavior.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 43 of its 70 references that are in the map. First posted on arXiv 3 Feb 2025.

Builds on

43 of 70 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024On the Expressiveness and Length Generalization of Selective State-Space Models on Regular Languages26 Dec
  2. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality31 May
  3. 2024The Hidden Attention of Mamba Models3 Mar
  4. 2024Theoretical Foundations of Deep Selective State-Space Models29 Feb
  5. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spaces1 Dec
  6. 2022Simplified State Space Layers for Sequence Modeling9 Aug
  7. 2021Efficiently Modeling Long Sequences with Structured State Spaces31 Oct

Other references · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2022Scalable Diffusion Models with Transformerscited by 5,931
  5. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  6. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  7. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  8. 2017Spectrally-normalized margin bounds for neural networkscited by 681
  9. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
  10. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 600
  11. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 418
  12. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
24 more
  1. 2017Information-theoretic analysis of generalization capability of learning algorithmscited by 299
  2. 2021User-friendly introduction to PAC-Bayes boundscited by 234
  3. 2020Reasoning About Generalization via Conditional Mutual Informationcited by 145
  4. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  5. 2018ListOps: A Diagnostic Dataset for Latent Tree Learningcited by 128
  6. 2020Sharpened Generalization Bounds based on Conditional Mutual Information and an Application to Noisy, Iterative Algorithmscited by 89
  7. 2019Data-dependent Sample Complexity of Deep Neural Networks via Lipschitz Augmentationcited by 71
  8. 2024The Expressive Capacity of State Space Models: A Formal Language Perspectivecited by 45
  9. 2019On Generalization Bounds of a Family of Recurrent Neural Networkscited by 42
  10. 2018Stabilizing Gradients for Deep Neural Networks via Efficient SVD Parameterizationcited by 35
  11. 2023StableSSM: Alleviating the Curse of Memory in State-space Models through Stable Reparameterizationcited by 29
  12. 2023Sequence Length Independent Norm-Based Generalization Bounds for Transformerscited by 28
  13. 2024Unlocking Tokens as Data Points for Generalization Bounds on Larger Language Modelscited by 25
  14. 2024Topological Generalization Bounds for Discrete-Time Stochastic Optimization Algorithmscited by 12
  15. 2024From Generalization Analysis to Optimization Designs for State Space Modelscited by 11
  16. 2024HOPE for a Robust Parameterization of Long-memory State Space Modelscited by 11
  17. 2024Solving Masked Jigsaw Puzzles with Diffusion Vision Transformerscited by 10
  18. 2024Autocorrelation Matters: Understanding the Role of Initialization Schemes for State Space Modelscited by 8
  19. 2024Generalization of Graph Neural Networks is Robust to Model Mismatchcited by 4
  20. 2023State space models can express n-gram languagescited by 3
  21. 2024On Generalisation Error Bounds for Transformerscited by 3
  22. 2024Length independent generalization bounds for deep SSM architectures via Rademacher contraction and stability constraintscited by 2
  23. 2024A Convex Relaxation Approach to Generalization Analysis for Parallel Positively Homogeneous Networkscited by 2
  24. 2024Generalization and Risk Bounds for Recurrent Neural Networkscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025Block-Biased Mamba for Long-Range Sequence Processingcited by 7
  2. 2026A Theoretical Analysis of Mamba's Training Dynamics: Filtering Relevant Features for Generalization in State Space Modelscited by 3
  3. 2025Local Stability and Region of Attraction Analysis for Neural Network Feedback Systems under Positivity Constraintscited by 2
  4. 2025Frequency Response Identification of Low-Order Systems: Finite-Sample Analysiscited by 0
  5. 2026Upper Generalization Bounds for Neural Oscillatorscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025On the Expressivity of Selective State-Space Layers: A Multivariate Polynomial Approach
  2. 2024An Empirical Study of Mamba-based Language Models
  3. 2024MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts
  4. 2025Rethinking Selectivity in State Space Models: A Minimal Predictive Sufficiency Approach
  5. 2022Simplifying and Understanding State Space Models with Diagonal Linear RNNs
  6. 2025Advancing Intelligent Sequence Modeling: Evolution, Trade-offs, and Applications of State-Space Architectures from S4 to Mamba
  7. 2024Sparse Mamba: Introducing Controllability, Observability, And Stability To Structural State Space Models
  8. 2025SeRpEnt: Selective Resampling for Expressive State Space Models