mlsanityData to 14 Oct 2026

On the Parameterization and Initialization of Diagonal State Space Models

arXiv 2206.11893 ↗First posted 23 Jun 2022cs.LGcited by 424

Not on the map. Why?

Cited by 150 in State-space sequence models (S4, Mamba), but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

State space models (SSM) have recently been shown to be very effective as a deep learning layer as a promising alternative to sequence models such as RNNs, CNNs, or Transformers. The first version to show this potential was the S4 model, which is particularly effective on tasks involving long-range dependencies by using a prescribed state matrix called the HiPPO matrix. While this has an interpretable mathematical mechanism for modeling long dependencies, it introduces a custom representation and algorithm that can be difficult to implement. On the other hand, a recent variant of S4 called DSS showed that restricting the state matrix to be fully diagonal can still preserve the performance of the original model when using a specific initialization based on approximating S4's matrix. This work seeks to systematically understand how to parameterize and initialize such diagonal state space models. While it follows from classical results that almost all SSMs have an equivalent diagonal form, we show that the initialization is critical for performance. We explain why DSS works mathematically, by showing that the diagonal restriction of S4's matrix surprisingly recovers the same kernel in the limit of infinite state dimension. We also systematically describe various design choices in parameterizing and computing diagonal SSMs, and perform a controlled empirical study ablating the effects of these choices. Our final model S4D is a simple diagonal version of S4 whose kernel computation requires just 2 lines of code and performs comparably to S4 in almost all settings, with state-of-the-art results for image, audio, and medical time-series domains, and averaging 85% on the Long Range Arena benchmark.

Key facts. Cited by 424 papers in the map (data through 14 Oct 2026). Builds on 17 of its 40 references that are in the map. First posted on arXiv 23 Jun 2022.

Builds on

17 of 40 references.

References · 17

  1. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  2. 2016Layer Normalizationcited by 5,669
  3. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  4. 2020GLU Variants Improve Transformercited by 1,804
  5. 2016Language Modeling with Gated Convolutional Networkscited by 702
  6. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  7. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 605
  8. 2020Neural Controlled Differential Equations for Irregular Time Seriescited by 506
  9. 2022Diagonal State Spaces are as Effective as Structured State Spacescited by 350
  10. 2022It's Raw! Audio Generation with State-Space Modelscited by 171
  11. 2020Neural Rough Differential Equations for Long Time Seriescited by 143
  12. 2021CKConv: Continuous Kernel Convolution For Sequential Datacited by 120
5 more
  1. 2020Lipschitz Recurrent Neural Networkscited by 89
  2. 2021FlexConv: Continuous Kernel Convolutions with Differentiable Kernel Sizescited by 66
  3. 2018Learning Longer-term Dependencies in RNNs with Auxiliary Lossescited by 58
  4. 2021UnICORNN: A recurrent model for learning very long time dependenciescited by 56
  5. 2018Trellis Networks for Sequence Modelingcited by 52

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 13; 2023: 38; 2024: 155; 2025: 122; 2026: 96.

424 papers.

Sort:

By area:

  1. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,056
  2. 2024VMamba: Visual State Space Modelcited by 969
  3. 2022Simplified State Space Layers for Sequence Modelingcited by 788
  4. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 499
  5. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 408
  6. 2022Long Range Language Modeling via Gated State Spacescited by 265
  7. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 242
  8. 2024LocalMamba: Visual State Space Model with Windowed Selective Scancited by 236
  9. 2024A Survey on Efficient Inference for Large Language Modelscited by 205
  10. 2022Mega: Moving Average Equipped Gated Attentioncited by 179
  11. 2024U-KAN Makes Strong Backbone for Medical Image Segmentation and Generationcited by 179
  12. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 174
38 more
  1. 2024PointMamba: A Simple State Space Model for Point Cloud Analysiscited by 167
  2. 2023Protein Design with Guided Discrete Diffusioncited by 157
  3. 2024EfficientVMamba: Atrous Selective Scan for Light Weight Visual Mambacited by 146
  4. 2022Liquid Structural State-Space Modelscited by 136
  5. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 131
  6. 2024HGRN2: Gated Linear RNNs with State Expansioncited by 127
  7. 2022How to Train Your HiPPO: State Space Models with Generalized Orthogonal Basis Projectionscited by 122
  8. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 122
  9. 2024Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modelingcited by 113
  10. 2024Motion Mamba: Efficient and Long Sequence Motion Generationcited by 112
  11. 2024Video Mamba Suite: State Space Model as a Versatile Alternative for Video Understandingcited by 108
  12. 2024Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Taskscited by 105
  13. 2023Hierarchically Gated Recurrent Neural Network for Sequence Modelingcited by 102
  14. 2024VL-Mamba: Exploring State Space Models for Multimodal Learningcited by 93
  15. 2024In-Context Language Learning: Architectures and Algorithmscited by 89
  16. 2026Mamba-3: Improved Sequence Modeling using State Space Principlescited by 89
  17. 2022What Makes Convolutional Models Great on Long Sequence Modeling?cited by 87
  18. 2024ZigMa: A DiT-style Zigzag Mamba Diffusion Modelcited by 84
  19. 2024RS-Mamba for Large Remote Sensing Image Dense Predictioncited by 81
  20. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  21. 2023Diffusion Models Without Attentioncited by 75
  22. 2024State Space Models for Event Camerascited by 75
  23. 2024Theoretical Foundations of Deep Selective State-Space Modelscited by 74
  24. 2025Test-time regression: a unifying framework for designing sequence models with associative memorycited by 74
  25. 2024Point Mamba: A Novel Point Cloud Backbone Based on State Space Model with Octree-Based Ordering Strategycited by 73
  26. 2024DiM: Diffusion Mamba for Efficient High-Resolution Image Synthesiscited by 72
  27. 2024Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challengescited by 64
  28. 2024ContiFormer: Continuous-Time Transformer for Irregular Time Series Modelingcited by 63
  29. 2023Simple Hardware-Efficient Long Convolutions for Sequence Modelingcited by 62
  30. 2024An Investigation of Incorporating Mamba for Speech Enhancementcited by 62
  31. 2024Mamba in Speech: Towards an Alternative to Self-Attentioncited by 60
  32. 2024MambaByte: Token-free Selective State Space Modelcited by 59
  33. 2024State Space Model for New-Generation Network Alternative to Transformers: A Surveycited by 59
  34. 2024Retinexmamba: Retinex-based Mamba for Low-light Image Enhancementcited by 59
  35. 2024Gamba: Marry Gaussian Splatting with Mamba for single view 3D reconstructioncited by 56
  36. 2023Effectively Modeling Time Series with Simple Discrete State Spacescited by 54
  37. 2022Pretraining Without Attentioncited by 51
  38. 2024MambaLLIE: Implicit Retinex-Aware Low Light Enhancement with Global-then-Local State Spacecited by 48

50 most cited of 424 All 424 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Autocorrelation Matters: Understanding the Role of Initialization Schemes for State Space Models
  2. 2024HiPPO-Prophecy: State-Space Models can Provably Learn Dynamical Systems in Context
  3. 2024Towards a theory of learning dynamics in deep state space models
  4. 2024Deep Learning-based Approaches for State Space Models: A Selective Review
  5. 2025From Layers to States: A State Space Model Perspective to Deep Neural Network Layer Dynamics
  6. 2023Spectral State Space Models
  7. 2022Learning neural state-space models: do we need a state estimator?
  8. 2025The Curious Case of In-Training Compression of State Space Models