mlsanityData to 7 Oct 2026

Mamba: Linear-Time Sequence Modeling with Selective State Spaces

arXiv 2312.00752 ↗First posted 1 Dec 2023cs.LG cs.AIcited by 4,616

On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language

Abstract (arXiv)

Foundation models, now powering most of the exciting applications in deep learning, are almost universally based on the Transformer architecture and its core attention module. Many subquadratic-time architectures such as linear attention, gated convolution and recurrent models, and structured state space models (SSMs) have been developed to address Transformers' computational inefficiency on long sequences, but they have not performed as well as attention on important modalities such as language. We identify that a key weakness of such models is their inability to perform content-based reasoning, and make several improvements. First, simply letting the SSM parameters be functions of the input addresses their weakness with discrete modalities, allowing the model to selectively propagate or forget information along the sequence length dimension depending on the current token. Second, even though this change prevents the use of efficient convolutions, we design a hardware-aware parallel algorithm in recurrent mode. We integrate these selective SSMs into a simplified end-to-end neural network architecture without attention or even MLP blocks (Mamba). Mamba enjoys fast inference (5× higher throughput than Transformers) and linear scaling in sequence length, and its performance improves on real data up to million-length sequences. As a general sequence model backbone, Mamba achieves state-of-the-art performance across several modalities such as language, audio, and genomics. On language modeling, our Mamba-3B model outperforms Transformers of the same size and matches Transformers twice its size, both in pretraining and downstream evaluation.

Key facts. Cited by 4,616 papers in the map (data through 7 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 27 of its 116 references that are in the map. First posted on arXiv 1 Dec 2023.

Builds on

27 of 116 references.

References · 27

  1. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  2. 2016Layer Normalizationcited by 5,579
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  5. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  7. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  8. 2020GLU Variants Improve Transformercited by 1,724
  9. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  10. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  11. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  12. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 741
15 more
  1. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  2. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  3. 2023LongNet: Scaling Transformers to 1,000,000,000 Tokenscited by 201
  4. 2017Searching for Activation Functionscited by 158
  5. 2021CKConv: Continuous Kernel Convolution For Sequential Datacited by 117
  6. 2021An Attention Free Transformercited by 111
  7. 2022The Devil in Linear Transformercited by 98
  8. 2023Scaling Transformer to 1M tokens and beyond with RMTcited by 97
  9. 2016Quasi-Recurrent Neural Networkscited by 87
  10. 2017Simple Recurrent Units for Highly Parallelizable Recurrencecited by 60
  11. 2022Efficient Long Sequence Modeling via State Space Augmented Transformercited by 37
  12. 2023Block-State Transformerscited by 24
  13. 2022Simplifying and Understanding State Space Models with Diagonal Linear RNNscited by 23
  14. 2023Focus Your Attention (with Adaptive IIR Filters)cited by 12
  15. 2023Time-Parameterized Convolutional Neural Networks for Irregularly Sampled Time Seriescited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 12; 2024: 1338; 2025: 1726; 2026: 1540.

4,616 papers.

Sort:

By area:

  1. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  2. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  3. 2024Vision Mamba: Efficient Visual Representation Learning with Bidirectional State Space Modelcited by 1,051
  4. 2024VMamba: Visual State Space Modelcited by 955
  5. 2024Simple and Effective Masked Diffusion Language Modelscited by 800
  6. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  7. 2025Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusioncited by 569
  8. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
  9. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  10. 2024Large Language Models: A Surveycited by 506
  11. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 490
  12. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
38 more
  1. 2024Sora: A Review on Background, Technology, Limitations, and Opportunities of Large Vision Modelscited by 467
  2. 2024YOLOv9: Learning What You Want to Learn Using Programmable Gradient Informationcited by 422
  3. 2024U-Mamba: Enhancing Long-range Dependency for Biomedical Image Segmentationcited by 412
  4. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  5. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  6. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  7. 2024ShortGPT: Layers in Large Language Models are More Redundant Than You Expectcited by 369
  8. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 360
  9. 2024MambaIR: A Simple Baseline for Image Restoration with State-Space Modelcited by 349
  10. 2024SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuningcited by 345
  11. 2024Learning to (Learn at Test Time): RNNs with Expressive Hidden Statescited by 329
  12. 2024VideoMamba: State Space Model for Efficient Video Understandingcited by 325
  13. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  14. 2024Titans: Learning to Memorize at Test Timecited by 303
  15. 2025Layer by Layer: Uncovering Hidden Representations in Language Modelscited by 293
  16. 2024VM-UNet: Vision Mamba UNet for Medical Image Segmentationcited by 286
  17. 2024Long-context LLMs Struggle with Long In-context Learningcited by 285
  18. 2025Memory in the Age of AI Agentscited by 285
  19. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Modelscited by 268
  20. 2024SegMamba: Long-range Sequential Modeling Mamba For 3D Medical Image Segmentationcited by 266
  21. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 266
  22. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  23. 2024LocalMamba: Visual State Space Model with Windowed Selective Scancited by 235
  24. 2024nnU-Net Revisited: A Call for Rigorous Validation in 3D Medical Image Segmentationcited by 235
  25. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 235
  26. 2024Foundation Models for Time Series Analysis: A Tutorial and Surveycited by 234
  27. 2024A Survey on Mixture of Experts in Large Language Modelscited by 233
  28. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 232
  29. 2025Mercury: Ultra-Fast Language Models Based on Diffusioncited by 225
  30. 2024MambaVision: A Hybrid Mamba-Transformer Vision Backbonecited by 222
  31. 2024A Survey on 3D Gaussian Splattingcited by 221
  32. 2024Retrieval Head Mechanistically Explains Long-Context Factualitycited by 219
  33. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 211
  34. 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 208
  35. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 200
  36. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  37. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 192
  38. 2024Deep Time Series Models: A Comprehensive Survey and Benchmarkcited by 191

50 most cited of 4,616 All 4,616 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025SeRpEnt: Selective Resampling for Expressive State Space Models
  2. 2021Efficiently Modeling Long Sequences with Structured State Spaces
  3. 2024MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts
  4. 2024Computation-Efficient Era: A Comprehensive Survey of State Space Models in Medical Image Analysis
  5. 2025Design Principles for Sequence Models via Coefficient Dynamics