mlsanityData to 7 Oct 2026

Diagonal State Spaces are as Effective as Structured State Spaces

arXiv 2203.14343 ↗First posted 27 Mar 2022cs.LG cs.CLcited by 350

On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language

Abstract (arXiv)

Modeling long range dependencies in sequential data is a fundamental step towards attaining human-level performance in many modalities such as text, vision, audio and video. While attention-based models are a popular and effective choice in modeling short-range interactions, their performance on tasks requiring long range reasoning has been largely inadequate. In an exciting result, Gu et al. (ICLR 2022) proposed the textitStructured State Space (S4) architecture delivering large gains over state-of-the-art models on several long-range tasks across various modalities. The core proposition of S4 is the parameterization of state matrices via a diagonal plus low rank structure, allowing efficient computation. In this work, we show that one can match the performance of S4 even without the low rank correction and thus assuming the state matrices to be diagonal. Our textitDiagonal State Space (DSS) model matches the performance of S4 on Long Range Arena tasks, speech classification on Speech Commands dataset, while being conceptually simpler and straightforward to implement.

Key facts. Cited by 350 papers in the map (data through 7 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 27 of its 34 references that are in the map. First posted on arXiv 27 Mar 2022.

Builds on

27 of 34 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Efficiently Modeling Long Sequences with Structured State Spaces31 Oct

Other references · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  6. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  7. 2020Longformer: The Long-Document Transformercited by 3,362
  8. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  9. 2020Reformer: The Efficient Transformercited by 1,658
  10. 2020Rethinking Attention with Performerscited by 1,584
  11. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  12. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
14 more
  1. 2020Efficient Transformers: A Surveycited by 784
  2. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
  3. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  4. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
  5. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 600
  6. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  7. 2022It's Raw! Audio Generation with State-Space Modelscited by 171
  8. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  9. 2019Blockwise Self-Attention for Long Document Understandingcited by 158
  10. 2020Kernel Operations on the GPU, with Autodiff, without Memory Overflowscited by 101
  11. 2021Memory-efficient Transformers via Top-k Attentioncited by 59
  12. 2021H-Transformer-1D: Fast One-Dimensional Hierarchical Attention for Sequencescited by 47
  13. 2020GMAT: Global Memory Augmentation for Transformerscited by 36
  14. 2021Value-aware Approximate Attentioncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 15; 2023: 43; 2024: 152; 2025: 83; 2026: 57.

350 papers.

Sort:

By area:

  1. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
  2. 2024VMamba: Visual State Space Modelcited by 955
  3. 2022Simplified State Space Layers for Sequence Modelingcited by 776
  4. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  5. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  6. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 418
  7. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  8. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  9. 2023Long-term Forecasting with TiDE: Time-series Dense Encodercited by 327
  10. 2022Long Range Language Modeling via Gated State Spacescited by 265
  11. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  12. 2024LocalMamba: Visual State Space Model with Windowed Selective Scancited by 235
38 more
  1. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  2. 2023Efficient Large Language Models: A Surveycited by 183
  3. 2024U-KAN Makes Strong Backbone for Medical Image Segmentation and Generationcited by 176
  4. 2024PointMamba: A Simple State Space Model for Point Cloud Analysiscited by 166
  5. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 155
  6. 2024EfficientVMamba: Atrous Selective Scan for Light Weight Visual Mambacited by 146
  7. 2022Liquid Structural State-Space Modelscited by 134
  8. 2024HGRN2: Gated Linear RNNs with State Expansioncited by 122
  9. 2022How to Train Your HiPPO: State Space Models with Generalized Orthogonal Basis Projectionscited by 121
  10. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 120
  11. 2024Motion Mamba: Efficient and Long Sequence Motion Generationcited by 111
  12. 2024Caduceus: Bi-Directional Equivariant Long-Range DNA Sequence Modelingcited by 108
  13. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  14. 2024VL-Mamba: Exploring State Space Models for Multimodal Learningcited by 93
  15. 2024The Hidden Attention of Mamba Modelscited by 88
  16. 2024ZigMa: A DiT-style Zigzag Mamba Diffusion Modelcited by 84
  17. 2024RS-Mamba for Large Remote Sensing Image Dense Predictioncited by 80
  18. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  19. 2026Mamba-3: Improved Sequence Modeling using State Space Principlescited by 76
  20. 2023Diffusion Models Without Attentioncited by 75
  21. 2024State Space Models for Event Camerascited by 75
  22. 2022Efficient Long-Text Understanding with Short-Text Modelscited by 71
  23. 2024Point Mamba: A Novel Point Cloud Backbone Based on State Space Model with Octree-Based Ordering Strategycited by 71
  24. 2024Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challengescited by 63
  25. 2023Simple Hardware-Efficient Long Convolutions for Sequence Modelingcited by 62
  26. 2023Understanding In-Context Learning in Transformers and LLMs by Learning to Learn Discrete Functionscited by 62
  27. 2024MiM-ISTD: Mamba-in-Mamba for Efficient Infrared Small Target Detectioncited by 59
  28. 2024State Space Model for New-Generation Network Alternative to Transformers: A Surveycited by 59
  29. 2024Retinexmamba: Retinex-based Mamba for Low-light Image Enhancementcited by 59
  30. 2024MambaByte: Token-free Selective State Space Modelcited by 58
  31. 2024Mamba in Speech: Towards an Alternative to Self-Attentioncited by 58
  32. 2024Longhorn: State Space Models are Amortized Online Learnerscited by 58
  33. 2023Effectively Modeling Time Series with Simple Discrete State Spacescited by 54
  34. 2023BAMBOO: A Comprehensive Benchmark for Evaluating Long Text Modeling Capacities of Large Language Modelscited by 53
  35. 2024Vision Mamba: A Comprehensive Survey and Taxonomycited by 53
  36. 2022Pretraining Without Attentioncited by 51
  37. 2022Efficient Movie Scene Detection using State-Space Transformerscited by 51
  38. 2024Vision-LSTM: xLSTM as Generic Vision Backbonecited by 51

50 most cited of 350 All 350 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Robustifying State-space Models for Long Sequences via Approximate Diagonalization
  2. 2023State Spaces Aren't Enough: Machine Translation Needs Attention
  3. 2024Enhanced Structured State Space Models via Grouped FIR Filtering and Attention Sink Mechanisms
  4. 2026State Rank Dynamics in Linear Attention LLMs
  5. 2025On Structured State-Space Duality
  6. 2026Benchmarking the Computational and Representational Efficiency of State Space Models against Transformers on Long-Context Dyadic Sessions
  7. 2026S4oP: Operator-level Pruning of Structured State Space Models for Resource-Constrained Devices
  8. 2025How Many Heads Make an SSM? A Unified Framework for Attention and State Space Models