mlsanityData to 14 Oct 2026

Provable Benefits of Complex Parameterizations for Structured State Space Models

arXiv 2410.14067 ↗First posted 17 Oct 2024cs.LG cs.AI cs.NEcited by 9

On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language

Abstract (arXiv)

Structured state space models (SSMs), the core engine behind prominent neural networks such as S4 and Mamba, are linear dynamical systems adhering to a specified structure, most notably diagonal. In contrast to typical neural network modules, whose parameterizations are real, SSMs often use complex parameterizations. Theoretically explaining the benefits of complex parameterizations for SSMs is an open problem. The current paper takes a step towards its resolution, by establishing formal gaps between real and complex diagonal SSMs. Firstly, we prove that while a moderate dimension suffices in order for a complex SSM to express all mappings of a real SSM, a much higher dimension is needed for a real SSM to express mappings of a complex SSM. Secondly, we prove that even if the dimension of a real SSM is high enough to express a given mapping, typically, doing so requires the parameters of the real SSM to hold exponentially large values, which cannot be learned in practice. In contrast, a complex SSM can express any given mapping with moderate parameter values. Experiments corroborate our theory, and suggest a potential extension of the theory that accounts for selectivity, a new architectural feature yielding state of the art performance.

Key facts. Cited by 9 papers in the map (data through 14 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 27 of its 49 references that are in the map. First posted on arXiv 17 Oct 2024.

Builds on

27 of 49 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024The Implicit Bias of Structured State Space Models Can Be Poisoned With Clean Labels14 Oct
  2. 2024DeciMamba: Exploring the Length Extrapolation Potential of Mamba20 Jun
  3. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality31 May
  4. 2024The Hidden Attention of Mamba Models3 Mar
  5. 2024Theoretical Foundations of Deep Selective State-Space Models29 Feb
  6. 2024BlackMamba: Mixture of Experts for State-Space Models1 Feb
  7. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spaces1 Dec
  8. 2021Efficiently Modeling Long Sequences with Structured State Spaces31 Oct

Other references · 19

  1. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  2. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 772
  3. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  4. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 499
  5. 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
  6. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 424
  7. 2018Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Networkcited by 420
  8. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 408
  9. 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 216
  10. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 155
  11. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 128
  12. 2024Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modelingcited by 113
7 more
  1. 2024Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Lengthcited by 46
  2. 2019On Generalization Bounds of a Family of Recurrent Neural Networkscited by 42
  3. 2023Universality of Linear Recurrences Followed by Non-linear Projections: Finite-Width Guarantees and Benefits of Complex Eigenvaluescited by 36
  4. 2024From Generalization Analysis to Optimization Designs for State Space Modelscited by 11
  5. 2022Learning Low Dimensional State Spaces with Overparameterized Recurrent Neural Netscited by 7
  6. 2019Universal Approximation of Input-Output Maps by Temporal Convolutional Netscited by 3
  7. 2024Evaluating Time-Series Training Dataset through Lens of Spectrum in Deep State Space Modelscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 4.

9 papers.

  1. 2025Selective Rotary Position Embeddingcited by 9
  2. 2025Block-Biased Mamba for Long-Range Sequence Processingcited by 8
  3. 2025Fast weight programming and linear transformers: from machine learning to neurobiologycited by 8
  4. 2025SiLIF: Structured State Space Model Dynamics and Parametrization for Spiking Neural Networkscited by 3
  5. 2026Spectral Filtering for Complex Linear Dynamical Systemscited by 2
  6. 2025Upper Approximation Bounds for Neural Oscillatorscited by 1
  7. 2026Jordan-RoPE: Non-Semisimple Relative Positional Encoding via Complex Jordan Blockscited by 1
  8. 2026State Propagation Also Satisfies: A Complex-Valued State-Space Model for Deterministic State Trackingcited by 1
  9. 2026A Memory Efficient Unified Algorithm for Online Learning of Linear Dynamical Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Structured state-space models are deep Wiener models
  2. 2024HiPPO-Prophecy: State-Space Models can Provably Learn Dynamical Systems in Context
  3. 2025The Curious Case of In-Training Compression of State Space Models
  4. 2024Autocorrelation Matters: Understanding the Role of Initialization Schemes for State Space Models
  5. 2024State Space Models as Foundation Models: A Control Theoretic Overview
  6. 2024Towards a theory of learning dynamics in deep state space models
  7. 2024State Space Models are Provably Comparable to Transformers in Dynamic Token Selection
  8. 2025Memory Determines Learning Direction: A Theory of Gradient-Based Optimization in State Space Models