mlsanityData to 7 Oct 2026

Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality

arXiv 2405.21060 ↗First posted 31 May 2024cs.LGcited by 1,283

On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language

Abstract (arXiv)

While Transformers have been the main architecture behind deep learning's success in language modeling, state-space models (SSMs) such as Mamba have recently been shown to match or outperform Transformers at small to medium scale. We show that these families of models are actually quite closely related, and develop a rich framework of theoretical connections between SSMs and variants of attention, connected through various decompositions of a well-studied class of structured semiseparable matrices. Our state space duality (SSD) framework allows us to design a new architecture (Mamba-2) whose core layer is an a refinement of Mamba's selective SSM that is 2-8X faster, while continuing to be competitive with Transformers on language modeling.

Key facts. Cited by 1,283 papers in the map (data through 7 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 43 of its 114 references that are in the map. First posted on arXiv 31 May 2024.

Builds on

43 of 114 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Jamba: A Hybrid Transformer-Mamba Language Model28 Mar
  2. 2024The Hidden Attention of Mamba Models3 Mar
  3. 2024Is Mamba Capable of In-Context Learning?5 Feb
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spaces1 Dec

Other references · 39

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  5. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  7. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  8. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  10. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  11. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  12. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 904
27 more
  1. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  2. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  3. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  4. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  5. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  6. 2021FNet: Mixing Tokens with Fourier Transformscited by 343
  7. 2024xLSTM: Extended Long Short-Term Memorycited by 320
  8. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 232
  9. 2024World Model on Million-Length Video And Language With Blockwise RingAttentioncited by 196
  10. 2017Searching for Activation Functionscited by 158
  11. 2024Eagle and Finch: RWKV with Matrix-Valued States and Dynamic Recurrencecited by 155
  12. 2024HGRN2: Gated Linear RNNs with State Expansioncited by 122
  13. 2024Zamba: A Compact 7B SSM Hybrid Modelcited by 118
  14. 2021An Attention Free Transformercited by 111
  15. 2022The Devil in Linear Transformercited by 98
  16. 2016Quasi-Recurrent Neural Networkscited by 87
  17. 2021NormFormer: Improved Transformer Pretraining with Extra Normalizationcited by 83
  18. 2024RecurrentGemma: Moving Past Transformers for Efficient Open Language Modelscited by 61
  19. 2017Simple Recurrent Units for Highly Parallelizable Recurrencecited by 60
  20. 2023Striped Attention: Faster Ring Attention for Causal Transformerscited by 52
  21. 2023GateLoop: Fully Data-Controlled Linear Recurrence for Sequence Modelingcited by 45
  22. 2023State-space Models with Layer-wise Nonlinearity are Universal Approximators with Exponential Decaying Memorycited by 38
  23. 2023TransNormerLLM: A Faster and Better Large Language Model with Improved TransNormercited by 34
  24. 2024Fewer Truncations Improve Language Modelingcited by 33
  25. 2024Dataset Decomposition: Faster LLM Training with Variable Sequence Length Curriculumcited by 25
  26. 2022Boosting Distributed Training Performance of the Unpadded BERT Modelcited by 15
  27. 2024Linear Transformers with Learnable Kernel Functions are Better In-Context Modelscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 229; 2025: 529; 2026: 525.

1,283 papers.

Sort:

By area:

  1. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  2. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  3. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  4. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  5. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 329
  6. 2024Titans: Learning to Memorize at Test Timecited by 303
  7. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 266
  8. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  9. 2025KernelBench: Can LLMs Write Efficient GPU Kernels?cited by 206
  10. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 188
  11. 2024An Empirical Study of Mamba-based Language Modelscited by 177
  12. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 169
38 more
  1. 2025Hierarchical Reasoning Modelcited by 167
  2. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  3. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  4. 2025TTT3R: 3D Reconstruction as Test-Time Trainingcited by 122
  5. 2025Test-Time Training Done Rightcited by 121
  6. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 120
  7. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 119
  8. 2024Long-LRM: Long-sequence Large Reconstruction Model for Wide-coverage Gaussian Splatscited by 111
  9. 2024LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecturecited by 100
  10. 2025One-Minute Video Generation with Test-Time Trainingcited by 100
  11. 2025Sparse VideoGen2: Accelerate Video Generation with Sparse Attention via Semantic-Aware Permutationcited by 100
  12. 2026Kimi K3: Open Frontier Intelligencecited by 99
  13. 2024Unlocking State-Tracking in Linear RNNs Through Negative Eigenvaluescited by 97
  14. 2024Hymba: A Hybrid-head Architecture for Small Language Modelscited by 95
  15. 2025NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Modelcited by 94
  16. 2024CATCH: Channel-Aware multivariate Time Series Anomaly Detection via Frequency Patchingcited by 91
  17. 2025Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Modelscited by 88
  18. 2024DRAMA: An Efficient End-to-end Motion Planner for Autonomous Driving with Mambacited by 87
  19. 2024StableAnimator: High-Quality Identity-Preserving Human Image Animationcited by 85
  20. 2025Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 85
  21. 2026Mamba-3: Improved Sequence Modeling using State Space Principlescited by 76
  22. 2024Gated Slot Attention for Efficient Linear-Time Sequence Modelingcited by 73
  23. 2025A Survey on Latent Reasoningcited by 71
  24. 2025DeltaProduct: Improving State-Tracking in Linear RNNs via Householder Productscited by 69
  25. 2024nGPT: Normalized Transformer with Representation Learning on the Hyperspherecited by 68
  26. 2024Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Modelscited by 66
  27. 2025Dynamic Chunking for End-to-End Hierarchical Sequence Modelingcited by 65
  28. 2025Radial Attention: O(nlog n) Sparse Attention with Energy Decay for Long Video Generationcited by 64
  29. 2025Test-time regression: a unifying framework for designing sequence models with associative memorycited by 63
  30. 2025It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimizationcited by 60
  31. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methodscited by 59
  32. 2025Long-Context State-Space Video World Modelscited by 59
  33. 2026Attention Residualscited by 57
  34. 2025NVIDIA Nemotron 3: Efficient and Open Intelligencecited by 54
  35. 2024Discrete Copula Diffusioncited by 53
  36. 2025End-to-End Test-Time Training for Long Contextcited by 53
  37. 2025Forgetting Transformer: Softmax Attention with a Forget Gatecited by 50
  38. 2026SANA-WM: Efficient Minute-Scale World Modeling with Hybrid Linear Diffusion Transformercited by 50

50 most cited of 1,283 All 1,283 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Tasks
  2. 2024MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts
  3. 2025On the Expressivity of Selective State-Space Layers: A Multivariate Polynomial Approach
  4. 2024State Space Models as Foundation Models: A Control Theoretic Overview
  5. 2026Global Divergence, Local Convergence: Representation Geometry in SSMs and Transformers
  6. 2023Block-State Transformers
  7. 2024State Space Model for New-Generation Network Alternative to Transformers: A Survey
  8. 2024Dual-path Mamba: Short and Long-term Bidirectional Selective Structured State Space Models for Speech Separation