mlsanityData to 14 Oct 2026

Theoretical Foundations of Deep Selective State-Space Models

arXiv 2402.19047 ↗First posted 29 Feb 2024cs.LG math.DScited by 74

On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language

Abstract (arXiv)

Structured state-space models (SSMs) such as S4, stemming from the seminal work of Gu et al., are gaining popularity as effective approaches for modeling sequential data. Deep SSMs demonstrate outstanding performance across a diverse set of domains, at a reduced training and inference cost compared to attention-based transformers. Recent developments show that if the linear recurrence powering SSMs allows for multiplicative interactions between inputs and hidden states (e.g. GateLoop, Mamba, GLA), then the resulting architecture can surpass in both in accuracy and efficiency attention-powered foundation models trained on text, at scales of billion parameters. In this paper, we give theoretical grounding to this recent finding using tools from Rough Path Theory: we show that when random linear recurrences are equipped with simple input-controlled transitions (selectivity mechanism), then the hidden state is provably a low-dimensional projection of a powerful mathematical object called the signature of the input -- capturing non-linear interactions between tokens at distinct timescales. Our theory not only motivates the success of modern selective state-space models such as Mamba but also provides a solid framework to understand the expressive power of future SSM variants.

Key facts. Cited by 74 papers in the map (data through 14 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 45 of its 77 references that are in the map. First posted on arXiv 29 Feb 2024.

Builds on

45 of 77 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Gated Linear Attention Transformers with Hardware-Efficient Training11 Dec
  2. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spaces1 Dec
  3. 2023Structured State Space Models for In-Context Reinforcement Learning7 Mar
  4. 2021Efficiently Modeling Long Sequences with Structured State Spaces31 Oct

Other references · 41

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,057
  4. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 827
  5. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  6. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 613
  7. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 605
  8. 2020Neural Controlled Differential Equations for Irregular Time Seriescited by 506
  9. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 499
  10. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 424
  11. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 408
  12. 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 216
29 more
  1. 2023Zoology: Measuring and Improving Recall in Efficient Language Modelscited by 204
  2. 2022It's Raw! Audio Generation with State-Space Modelscited by 171
  3. 2020Neural Rough Differential Equations for Long Time Seriescited by 143
  4. 2017Parallelizing Linear Recurrent Neural Nets Over Sequence Lengthcited by 127
  5. 2016A Primer on the Signature Method in Machine Learningcited by 104
  6. 2022What Makes Convolutional Models Great on Long Sequence Modeling?cited by 87
  7. 2019Deep Signature Transformscited by 75
  8. 2020The Signature Kernel is the solution of a Goursat PDEcited by 58
  9. 2018Can recurrent neural networks warp time?cited by 55
  10. 2022Pretraining Without Attentioncited by 51
  11. 2021Neural Stochastic PDEs: Resolution-Invariant Learning of Continuous Spatiotemporal Dynamicscited by 47
  12. 2023GateLoop: Fully Data-Controlled Linear Recurrence for Sequence Modelingcited by 46
  13. 2022S4ND: Modeling Images and Videos as Multidimensional Signals Using State Spacescited by 41
  14. 2023State-space Models with Layer-wise Nonlinearity are Universal Approximators with Exponential Decaying Memorycited by 39
  15. 2023Non-adversarial training of Neural SDEs with signature kernel scorescited by 37
  16. 2019On the Computational Power of RNNscited by 35
  17. 2024Log Neural Controlled Differential Equations: The Lie Brackets Make a Differencecited by 35
  18. 2020Distribution Regression for Sequential Datacited by 31
  19. 2019Embedding and learning with signaturescited by 29
  20. 2021Higher Order Kernel Mean Embeddings to Capture Filtrations of Stochastic Processescited by 27
  21. 2023Neural signature kernels as infinite-width-depth-limits of controlled ResNetscited by 25
  22. 2020Discrete-time signatures and randomness in reservoir computingcited by 23
  23. 2021Framing RNN as a kernel method: A neural ODE approachcited by 23
  24. 2021SigGPDE: Scaling Sparse Gaussian Processes on Sequential Datacited by 22
  25. 2024Signature Kernel Conditional Independence Tests in Causal Discovery for Stochastic Processescited by 18
  26. 2021SK-Tree: a systematic malware detection algorithm on streaming trees via the signature kernelcited by 16
  27. 2023A Neural RDE approach for continuous-time non-Markovian stochastic control problemscited by 14
  28. 2023New directions in the applications of rough path theorycited by 12
  29. 2019Universal Approximation of Input-Output Maps by Temporal Convolutional Netscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 19; 2025: 31; 2026: 24.

74 papers.

Sort:

By area:

  1. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 159
  2. 2024Unlocking State-Tracking in Linear RNNs Through Negative Eigenvaluescited by 101
  3. 2025DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Productscited by 75
  4. 2024State Space Model for New-Generation Network Alternative to Transformers: A Surveycited by 59
  5. 2025PaTH Attention: Position Encoding via Accumulating Householder Transformationscited by 34
  6. 2024Understanding the differences in Foundation Models: Attention, State Space Models, and Recurrent Neural Networkscited by 26
  7. 2025Structured Linear CDEs: Maximally Expressive and Parallel-in-Time Sequence Modelscited by 18
  8. 2024Rough Transformers: Lightweight and Continuous Time Series Modelling through Signature Patchingcited by 16
  9. 2025Structured Sparse Transition Matrices to Enable State Tracking in State-Space Modelscited by 16
  10. 2025ParaRNN: Unlocking Parallel Training of Nonlinear RNNs for Large Language Modelscited by 15
  11. 2024Deep Continuous-Time State-Space Models for Marked Event Sequencescited by 11
  12. 2025From Markov to Laplace: How Mamba In-Context Learns Markov Chainscited by 11
38 more
  1. 2025Message-Passing State-Space Models: Improving Graph Learning with Modern Sequence Modelingcited by 10
  2. 2024Provable Benefits of Complex Parameterizations for Structured State Space Modelscited by 9
  3. 2024Autocorrelation Matters: Understanding the Role of Initialization Schemes for State Space Modelscited by 9
  4. 2025Advancing Intelligent Sequence Modeling: Evolution, Trade-offs, and Applications of State-Space Architectures from S4 to Mambacited by 9
  5. 2025Selective Rotary Position Embeddingcited by 9
  6. 2025Block-Biased Mamba for Long-Range Sequence Processingcited by 8
  7. 2025Fast weight programming and linear transformers: from machine learning to neurobiologycited by 8
  8. 2025Revisiting associative recall in modern recurrent modelscited by 8
  9. 2026Kalman Linear Attention: Parallel Bayesian Filtering For Efficient Language Modelling and State Trackingcited by 8
  10. 2025Understanding Input Selectivity in Mamba: Impact on Approximation Power, Memorization, and Associative Recall Capacitycited by 7
  11. 2024State Space Models are Provably Comparable to Transformers in Dynamic Token Selectioncited by 6
  12. 2024Sparse Signature Coefficient Recovery via Kernelscited by 6
  13. 2025Generalization Error Analysis for Selective State-Space Models Through the Lens of Attentioncited by 5
  14. 2025Fixed-Point RNNs: Interpolating from Diagonal to Densecited by 5
  15. 2024SigDiffusions: Score-Based Diffusion Models for Time Series via Log-Signature Embeddingscited by 4
  16. 2024Deep Learning-based Approaches for State Space Models: A Selective Reviewcited by 4
  17. 2025Random Feature Representation Boostingcited by 4
  18. 2025On the Expressivity of Selective State-Space Layers: A Multivariate Polynomial Approachcited by 4
  19. 2025ParallelFlow: Parallelizing Linear Transformers via Flow Discretizationcited by 4
  20. 2024NIMBA: Towards Robust and Principled Processing of Point Clouds With SSMscited by 3
  21. 2025Permutation Equivariant Neural Controlled Differential Equations for Dynamic Graph Representation Learningcited by 3
  22. 2026Attention to Mamba: A Recipe for Cross-Architecture Distillationcited by 3
  23. 2024Length independent generalization bounds for deep SSM architectures via Rademacher contraction and stability constraintscited by 2
  24. 2024Mathematical Formalism for Memory Compression in Selective State Space Modelscited by 2
  25. 2024Demystifying the Token Dynamics of Deep Selective State Space Modelscited by 2
  26. 2025Technologies on Effectiveness and Efficiency: A Survey of State Spaces Modelscited by 2
  27. 2025Regularity and Stability Properties of Selective SSMs with Discontinuous Gatingcited by 2
  28. 2025Time-Scale Coupling Between States and Parameters in Recurrent Neural Networkscited by 2
  29. 2025Context-Selective State Space Models: Feedback is All You Needcited by 2
  30. 2026On the "Induction Bias" in Sequence Modelscited by 2
  31. 2024On the low-shot transferability of [V]-Mambacited by 1
  32. 2024BadVim: Unveiling Backdoor Threats in Visual State Space Modelcited by 1
  33. 2025The Effect of Depth on the Expressivity of Deep Linear State-Space Modelscited by 1
  34. 2025Design Principles for Sequence Models via Coefficient Dynamicscited by 1
  35. 2025Upper Approximation Bounds for Neural Oscillatorscited by 1
  36. 2026Retrievit: In-context Retrieval Capabilities of Transformers, State Space Models, and Hybrid Architecturescited by 1
  37. 2026Why Depth Matters in Parallelizable Sequence Models: A Lie Algebraic Viewcited by 1
  38. 2026Understanding the Theoretical Foundations of Deep Neural Networks through Differential Equationscited by 1

50 most cited of 74 All 74 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Rethinking Selectivity in State Space Models: A Minimal Predictive Sufficiency Approach
  2. 2025SeRpEnt: Selective Resampling for Expressive State Space Models
  3. 2026The Attention Within: Consensus Dynamics in Selective State Space Models
  4. 2026A Theoretical Analysis of Mamba's Training Dynamics: Filtering Relevant Features for Generalization in State Space Models
  5. 2024On the Expressiveness and Length Generalization of Selective State-Space Models on Regular Languages
  6. 2024State Space Models as Foundation Models: A Control Theoretic Overview
  7. 2024Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothing
  8. 2026TIDES: Implicit Time-Awareness in Selective State Space Models