# Provable Benefits of Complex Parameterizations for Structured State Space Models arXiv: https://arxiv.org/abs/2410.14067 ยท HTML: https://mlsanity.com/paper/2410.14067/ ## Key facts - Cited by 9 papers in the map (data through 14 Oct 2026). - Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). - Builds on 27 of its 49 references that are in the map. - First posted on arXiv 17 Oct 2024. ## Placed in - State-space sequence models (S4, Mamba) / [Core SSM architectures for language](https://mlsanity.com/ssm/core-ssm-architectures-for-language/) ## Builds on (most cited first, 27 of 27) - [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016) - [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023) - [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021) - [Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality](https://mlsanity.com/paper/2405.21060/) (2024) - [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019) - [Long Range Arena: A Benchmark for Efficient Transformers](https://mlsanity.com/paper/2011.04006/) (2020) - [Hungry Hungry Hippos: Towards Language Modeling with State Space Models](https://mlsanity.com/paper/2212.14052/) (2022) - [Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability](https://mlsanity.com/paper/2103.00065/) (2021) - [On the Parameterization and Initialization of Diagonal State Space Models](https://mlsanity.com/paper/2206.11893/) (2022) - [Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network](https://mlsanity.com/paper/1808.03314/) (2018) - [Resurrecting Recurrent Neural Networks for Long Sequences](https://mlsanity.com/paper/2303.06349/) (2023) - [Repeat After Me: Transformers are Better than State Space Models at Copying](https://mlsanity.com/paper/2402.01032/) (2024) - [Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability](https://mlsanity.com/paper/2209.15594/) (2022) - [Understanding Gradient Descent on Edge of Stability in Deep Learning](https://mlsanity.com/paper/2205.09745/) (2022) - [Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modeling](https://mlsanity.com/paper/2403.03234/) (2024) - [The Hidden Attention of Mamba Models](https://mlsanity.com/paper/2403.01590/) (2024) - [Theoretical Foundations of Deep Selective State-Space Models](https://mlsanity.com/paper/2402.19047/) (2024) - [DeciMamba: Exploring the Length Extrapolation Potential of Mamba](https://mlsanity.com/paper/2406.14528/) (2024) - [Megalodon: Efficient LLM Pretraining and Inference with Unlimited Context Length](https://mlsanity.com/paper/2404.08801/) (2024) - [On Generalization Bounds of a Family of Recurrent Neural Networks](https://mlsanity.com/paper/1910.12947/) (2019) - [BlackMamba: Mixture of Experts for State-Space Models](https://mlsanity.com/paper/2402.01771/) (2024) - [Universality of Linear Recurrences Followed by Non-linear Projections: Finite-Width Guarantees and Benefits of Complex Eigenvalues](https://mlsanity.com/paper/2307.11888/) (2023) - [From Generalization Analysis to Optimization Designs for State Space Models](https://mlsanity.com/paper/2405.02670/) (2024) - [Learning Low Dimensional State Spaces with Overparameterized Recurrent Neural Nets](https://mlsanity.com/paper/2210.14064/) (2022) - [Universal Approximation of Input-Output Maps by Temporal Convolutional Nets](https://mlsanity.com/paper/1906.09211/) (2019) - [The Implicit Bias of Structured State Space Models Can Be Poisoned With Clean Labels](https://mlsanity.com/paper/2410.10473/) (2024) - [Evaluating Time-Series Training Dataset through Lens of Spectrum in Deep State Space Models](https://mlsanity.com/paper/2408.16261/) (2024) ## Built on by (most cited first, 9 of 9) - [Selective Rotary Position Embedding](https://mlsanity.com/paper/2511.17388/) (2025) - [Block-Biased Mamba for Long-Range Sequence Processing](https://mlsanity.com/paper/2505.09022/) (2025) - [Fast weight programming and linear transformers: from machine learning to neurobiology](https://mlsanity.com/paper/2508.08435/) (2025) - [SiLIF: Structured State Space Model Dynamics and Parametrization for Spiking Neural Networks](https://mlsanity.com/paper/2506.06374/) (2025) - [Spectral Filtering for Complex Linear Dynamical Systems](https://mlsanity.com/paper/2601.22400/) (2026) - [Upper Approximation Bounds for Neural Oscillators](https://mlsanity.com/paper/2512.01015/) (2025) - [Jordan-RoPE: Non-Semisimple Relative Positional Encoding via Complex Jordan Blocks](https://mlsanity.com/paper/2605.04217/) (2026) - [State Propagation Also Satisfies: A Complex-Valued State-Space Model for Deterministic State Tracking](https://mlsanity.com/paper/2608.03425/) (2026) - [A Memory Efficient Unified Algorithm for Online Learning of Linear Dynamical Systems](https://mlsanity.com/paper/2607.02050/) (2026) ## Abstract (arXiv) Structured state space models (SSMs), the core engine behind prominent neural networks such as S4 and Mamba, are linear dynamical systems adhering to a specified structure, most notably diagonal. In contrast to typical neural network modules, whose parameterizations are real, SSMs often use complex parameterizations. Theoretically explaining the benefits of complex parameterizations for SSMs is an open problem. The current paper takes a step towards its resolution, by establishing formal gaps between real and complex diagonal SSMs. Firstly, we prove that while a moderate dimension suffices in order for a complex SSM to express all mappings of a real SSM, a much higher dimension is needed for a real SSM to express mappings of a complex SSM. Secondly, we prove that even if the dimension of a real SSM is high enough to express a given mapping, typically, doing so requires the parameters of the real SSM to hold exponentially large values, which cannot be learned in practice. In contrast, a complex SSM can express any given mapping with moderate parameter values. Experiments corroborate our theory, and suggest a potential extension of the theory that accounts for selectivity, a new architectural feature yielding state of the art performance.