mlsanityData to 14 Oct 2026

From Generalization Analysis to Optimization Designs for State Space Models

arXiv 2405.02670 ↗First posted 4 May 2024cs.LGcited by 11

Not on the map. Why?

Abstract (arXiv)

A State Space Model (SSM) is a foundation model in time series analysis, which has recently been shown as an alternative to transformers in sequence modeling. In this paper, we theoretically study the generalization of SSMs and propose improvements to training algorithms based on the generalization results. Specifically, we give a textitdata-dependent generalization bound for SSMs, showing an interplay between the SSM parameters and the temporal dependencies of the training sequences. Leveraging the generalization bound, we (1) set up a scaling rule for model initialization based on the proposed generalization measure, which significantly improves the robustness of the output value scales on SSMs to different temporal patterns in the sequence data; (2) introduce a new regularization method for training SSMs to enhance the generalization performance. Numerical results are conducted to validate our results.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 24 of its 44 references that are in the map. First posted on arXiv 4 May 2024.

Builds on

24 of 44 references.

References · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  4. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  5. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  6. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,367
  7. 2022Simplified State Space Layers for Sequence Modelingcited by 788
  8. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  9. 2019Enhancing the Locality and Breaking the Memory Bottleneck of Transformer on Time Series Forecastingcited by 607
  10. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 499
  11. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 424
  12. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 408
12 more
  1. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 393
  2. 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
  3. 2022Long Range Language Modeling via Gated State Spacescited by 265
  4. 2022It's Raw! Audio Generation with State-Space Modelscited by 171
  5. 2018ListOps: A Diagnostic Dataset for Latent Tree Learningcited by 129
  6. 2022How to Train Your HiPPO: State Space Models with Generalized Orthogonal Basis Projectionscited by 122
  7. 2018Learning long-range spatial dependencies with horizontal gated-recurrent unitscited by 98
  8. 2019On Generalization Bounds of a Family of Recurrent Neural Networkscited by 42
  9. 2018Stabilizing Gradients for Deep Neural Networks via Efficient SVD Parameterizationcited by 35
  10. 2020On the Curse of Memory in Recurrent Neural Networks: Approximation and Optimization Analysiscited by 27
  11. 2019Can SGD Learn Recurrent Neural Networks with Provable Generalization?cited by 17
  12. 2023Inverse Approximation Theory for Nonlinear Recurrent Neural Networkscited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 4; 2026: 2.

11 papers.

  1. 2024Tuning Frequency Bias of State Space Modelscited by 17
  2. 2024HOPE for a Robust Parameterization of Long-memory State Space Modelscited by 12
  3. 2024Provable Benefits of Complex Parameterizations for Structured State Space Modelscited by 9
  4. 2025Block-Biased Mamba for Long-Range Sequence Processingcited by 8
  5. 2025Generalization Error Analysis for Selective State-Space Models Through the Lens of Attentioncited by 5
  6. 2025On the Expressivity of Selective State-Space Layers: A Multivariate Polynomial Approachcited by 4
  7. 2024Length independent generalization bounds for deep SSM architectures via Rademacher contraction and stability constraintscited by 2
  8. 2024Demystifying the Token Dynamics of Deep Selective State Space Modelscited by 2
  9. 2026Analysis of Long Range Dependency Understanding in State Space Modelscited by 1
  10. 2025Aligning Inductive Bias for Data-Efficient Generalization in State Space Modelscited by 0
  11. 2026Upper Generalization Bounds for Neural Oscillatorscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Time-SSM: Simplifying and Unifying State Space Models for Time Series Forecasting
  2. 2024Autocorrelation Matters: Understanding the Role of Initialization Schemes for State Space Models
  3. 2024Deep Learning-based Approaches for State Space Models: A Selective Review
  4. 2023Effectively Modeling Time Series with Simple Discrete State Spaces
  5. 2025A Generalizable Physics-Enhanced State Space Model for Long-Term Dynamics Forecasting in Complex Environments
  6. 2025MS-SSM: A Multi-Scale State Space Model for Efficient Sequence Modeling
  7. 2026A Simple State Space Model Excels at Multivariate Time Series Classification
  8. 2024State Space Models as Foundation Models: A Control Theoretic Overview