mlsanityData to 14 Oct 2026

Length independent generalization bounds for deep SSM architectures via Rademacher contraction and stability constraints

arXiv 2405.20278 ↗First posted 30 May 2024cs.LG cs.AI stat.MLcited by 2

Not on the map. Why?

Abstract (arXiv)

Many state-of-the-art models trained on long-range sequences, for example S4, S5 or LRU, are made of sequential blocks combining State-Space Models (SSMs) with neural networks. In this paper we provide a PAC bound that holds for these kind of architectures with stable SSM blocks and does not depend on the length of the input sequence. Imposing stability of the SSM blocks is a standard practice in the literature, and it is known to help performance. Our results provide a theoretical justification for the use of stable SSM blocks as the proposed PAC bound decreases as the degree of stability of the SSM blocks increases.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 51 of its 83 references that are in the map. First posted on arXiv 30 May 2024.

Builds on

51 of 83 references.

References · 51

  1. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  2. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  3. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  4. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,304
  5. 2022Simplified State Space Layers for Sequence Modelingcited by 788
  6. 2016Language Modeling with Gated Convolutional Networkscited by 702
  7. 2017Spectrally-normalized margin bounds for neural networkscited by 686
  8. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 608
  9. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 499
  10. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 424
  11. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 408
  12. 2017Size-Independent Sample Complexity of Neural Networkscited by 290
38 more
  1. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  2. 2016A vector-contraction inequality for Rademacher complexitiescited by 142
  3. 2022How to Train Your HiPPO: State Space Models with Generalized Orthogonal Basis Projectionscited by 122
  4. 2017Fisher-Rao Metric, Geometry, and Complexity of Neural Networkscited by 100
  5. 2024Theoretical Foundations of Deep Selective State-Space Modelscited by 74
  6. 2023Advancing Transformer Architecture in Long-Context Large Language Models: A Comprehensive Surveycited by 72
  7. 2019Data-dependent Sample Complexity of Deep Neural Networks via Lipschitz Augmentationcited by 71
  8. 2019Finite Sample Analysis of Stochastic System Identificationcited by 67
  9. 2019Generalization Guarantees for Neural Networks via Harnessing the Low-rank Structure of the Jacobiancited by 67
  10. 2024State Space Model for New-Generation Network Alternative to Transformers: A Surveycited by 59
  11. 2019Learning Linear Dynamical Systems with Semi-Parametric Least Squarescited by 58
  12. 2020Logarithmic Regret Bound in Partially Observable Linear Dynamical Systemscited by 46
  13. 2019On Generalization Bounds of a Family of Recurrent Neural Networkscited by 42
  14. 2020Logarithmic Regret for Adversarial Online Controlcited by 41
  15. 2020Non-asymptotic Identification of Linear Dynamical Systems Using Multiple Trajectoriescited by 38
  16. 2023Universality of Linear Recurrences Followed by Non-linear Projections: Finite-Width Guarantees and Benefits of Complex Eigenvaluescited by 36
  17. 2018Stabilizing Gradients for Deep Neural Networks via Efficient SVD Parameterizationcited by 35
  18. 2023LipsFormer: Introducing Lipschitz Continuity to Vision Transformerscited by 34
  19. 2023Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priorscited by 34
  20. 2022Learning with little mixingcited by 33
  21. 2023StableSSM: Alleviating the Curse of Memory in State-space Models through Stable Reparameterizationcited by 30
  22. 2023Sequence Length Independent Norm-Based Generalization Bounds for Transformerscited by 29
  23. 2022Single Trajectory Nonparametric Learning of Nonlinear Dynamicscited by 26
  24. 2022Learning from many trajectoriescited by 24
  25. 2023Laughing Hyena Distillery: Extracting Compact Recurrences From Convolutionscited by 24
  26. 2021Framing RNN as a kernel method: A neural ODE approachcited by 23
  27. 2022Online PAC-Bayes Learningcited by 22
  28. 2023Generalization bounds for neural ordinary differential equations and deep residual networkscited by 21
  29. 2024State-Free Inference of State-Space Models: The Transfer Function Approachcited by 18
  30. 2024On the Resurgence of Recurrent Models for Long Sequences -- Survey and Research Opportunities in the Transformer Eracited by 17
  31. 2020Learning Partially Known Stochastic Dynamics with Empirical PAC Bayescited by 16
  32. 2022On Rademacher Complexity-based Generalization Bounds for Deep Learningcited by 16
  33. 2019Improved PAC-Bayesian Bounds for Linear Regressioncited by 12
  34. 2020Learning Recurrent Neural Net Models of Nonlinear Systemscited by 11
  35. 2024From Generalization Analysis to Optimization Designs for State Space Modelscited by 11
  36. 2024Sharp Rates in Dependent Learning Theory: Avoiding Sample Size Deflation for the Square Losscited by 10
  37. 2021Generalization Error Bounds on Deep Learning with Markov Datasetscited by 9
  38. 2024Rademacher Complexity of Neural ODEs via Chen-Fliess Seriescited by 5

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Generalization Error Analysis for Selective State-Space Models Through the Lens of Attentioncited by 5
  2. 2026Upper Generalization Bounds for Neural Oscillatorscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024State Space Models as Foundation Models: A Control Theoretic Overview
  2. 2025A Deep State-Space Model Compression Method using Upper Bound on Output Error
  3. 2025The Effect of Depth on the Expressivity of Deep Linear State-Space Models
  4. 2025L2RU: a Structured State Space Model with prescribed L2-bound
  5. 2025The Curious Case of In-Training Compression of State Space Models
  6. 2026LipSSM: Structurally Lipschitz-Bounded Cascaded State-Space Model via Metric Transfer between Consecutive SSM Layers
  7. 2024HOPE for a Robust Parameterization of Long-memory State Space Models
  8. 2024On Provable Length and Compositional Generalization