mlsanityData to 7 Oct 2026

An Empirical Study of Mamba-based Language Models

arXiv 2406.07887 ↗First posted 12 Jun 2024cs.LG cs.CLcited by 177

On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language

also in 1 more
Abstract (arXiv)

Selective state-space models (SSMs) like Mamba overcome some of the shortcomings of Transformers, such as quadratic computational complexity with sequence length and large inference-time memory requirements from the key-value cache. Moreover, recent studies have shown that SSMs can match or exceed the language modeling capabilities of Transformers, making them an attractive alternative. In a controlled setting (e.g., same data), however, studies so far have only presented small scale experiments comparing SSMs to Transformers. To understand the strengths and weaknesses of these architectures at larger scales, we present a direct comparison between 8B-parameter Mamba, Mamba-2, and Transformer models trained on the same datasets of up to 3.5T tokens. We also compare these models to a hybrid architecture consisting of 43% Mamba-2, 7% attention, and 50% MLP layers (Mamba-2-Hybrid). Using a diverse set of tasks, we answer the question of whether Mamba models can match Transformers at larger training budgets. Our results show that while pure SSMs match or exceed Transformers on many tasks, they lag behind Transformers on tasks which require strong copying or in-context learning abilities (e.g., 5-shot MMLU, Phonebook) or long-context reasoning. In contrast, we find that the 8B Mamba-2-Hybrid exceeds the 8B Transformer on all 12 standard tasks we evaluated (+2.65 points on average) and is predicted to be up to 8x faster when generating tokens at inference time. To validate long-context capabilities, we provide additional experiments evaluating variants of the Mamba-2-Hybrid and Transformer extended to support 16K, 32K, and 128K sequences. On an additional 23 long-context tasks, the hybrid model continues to closely match or exceed the Transformer on average. To enable further study, we release the checkpoints as well as the code used to train our models as part of NVIDIA's Megatron-LM project.

Key facts. Cited by 177 papers in the map (data through 7 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 48 of its 53 references that are in the map. First posted on arXiv 12 Jun 2024.

Builds on

48 of 53 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality31 May
  2. 2024Jamba: A Hybrid Transformer-Mamba Language Model28 Mar
  3. 2024Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Tasks6 Feb
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spaces1 Dec
  5. 2021Efficiently Modeling Long Sequences with Structured State Spaces31 Oct

Other references · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2016Layer Normalizationcited by 5,579
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  9. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  11. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  12. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
31 more
  1. 2023Mistral 7Bcited by 3,065
  2. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  3. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  4. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  5. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  6. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  8. 2020GLU Variants Improve Transformercited by 1,724
  9. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  10. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  11. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  12. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  13. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,270
  14. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,259
  15. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 893
  16. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  17. 2020Efficient Transformers: A Surveycited by 784
  18. 2019Latent Retrieval for Weakly Supervised Open Domain Question Answeringcited by 691
  19. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  20. 2017The NarrativeQA Reading Comprehension Challengecited by 590
  21. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  22. 2023Effective Long-Context Scaling of Foundation Modelscited by 301
  23. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 300
  24. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 232
  25. 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 208
  26. 2023Zoology: Measuring and Improving Recall in Efficient Language Modelscited by 188
  27. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  28. 2024Zamba: A Compact 7B SSM Hybrid Modelcited by 118
  29. 2024Nemotron-4 15B Technical Reportcited by 36
  30. 2023Diagonal State Space Augmented Transformers for Speech Recognitioncited by 25
  31. 2023Block-State Transformerscited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 40; 2025: 89; 2026: 48.

177 papers.

Sort:

By area:

  1. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  2. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  3. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 169
  4. 2025Cosmos-Reason1: From Physical Common Sense To Embodied Reasoningcited by 159
  5. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  6. 2025TTT3R: 3D Reconstruction as Test-Time Trainingcited by 122
  7. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 120
  8. 2024Long-LRM: Long-sequence Large Reconstruction Model for Wide-coverage Gaussian Splatscited by 111
  9. 2024Hymba: A Hybrid-head Architecture for Small Language Modelscited by 95
  10. 2025NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Modelcited by 94
  11. 2025Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Modelscited by 88
  12. 2026Mamba-3: Improved Sequence Modeling using State Space Principlescited by 76
38 more
  1. 2024Gated Slot Attention for Efficient Linear-Time Sequence Modelingcited by 73
  2. 2025Dynamic Chunking for End-to-End Hierarchical Sequence Modelingcited by 65
  3. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methodscited by 59
  4. 2025NVIDIA Nemotron 3: Efficient and Open Intelligencecited by 54
  5. 2025Forgetting Transformer: Softmax Attention with a Forget Gatecited by 50
  6. 2024Jamba-1.5: Hybrid Transformer-Mamba Models at Scalecited by 48
  7. 2025STORM: Token-Efficient Long Video Understanding for Multimodal LLMscited by 46
  8. 2024Falcon Mamba: The First Competitive Attention-free 7B Language Modelcited by 43
  9. 2024LinGen: Towards High-Resolution Minute-Length Text-to-Video Generation with Linear Computational Complexitycited by 41
  10. 2025Baichuan-M1: Pushing the Medical Capability of Large Language Modelscited by 41
  11. 2024A Survey of Mambacited by 39
  12. 2025Jet-Nemotron: Efficient Language Model with Post Neural Architecture Searchcited by 39
  13. 2025A Systematic Analysis of Hybrid Linear Attentioncited by 38
  14. 2025LongMamba: Enhancing Mamba's Long Context Capabilities via Training-Free Receptive Field Enlargementcited by 36
  15. 2025Llamba: Scaling Distilled Recurrent Models for Efficient Language Processingcited by 35
  16. 2024Marconi: Prefix Caching for the Era of Hybrid LLMscited by 29
  17. 2024Speech Slytherin: Examining the Performance and Efficiency of Mamba for Speech Separation, Recognition, and Synthesiscited by 22
  18. 2024The Zamba2 Suite: Technical Reportcited by 20
  19. 2025GENERator: A Long-Context Generative Genomic Foundation Modelcited by 20
  20. 2025Understanding and Improving Length Generalization in Recurrent Modelscited by 19
  21. 2025Zebra-Llama: Towards Extremely Efficient Hybrid Modelscited by 18
  22. 2024Understanding and Mitigating Bottlenecks of State Space Models through the Lens of Recency and Over-smoothingcited by 17
  23. 2025xLSTM 7B: A Recurrent LLM for Fast and Efficient Inferencecited by 17
  24. 2024Mimetic Initialization Helps State Space Models Learn to Recallcited by 15
  25. 2025Understanding the Skill Gap in Recurrent Language Models: The Role of the Gather-and-Aggregate Mechanismcited by 15
  26. 2025Structured Sparse Transition Matrices to Enable State Tracking in State-Space Modelscited by 15
  27. 2025Nemotron-Flash: Towards Latency-Optimal Hybrid Small Language Modelscited by 15
  28. 2024DyGMamba: Efficiently Modeling Long-Term Temporal Dependency on Continuous-Time Dynamic Graphs with State Space Modelscited by 14
  29. 2025Quamba2: A Robust and Scalable Post-training Quantization Framework for Selective State Space Modelscited by 14
  30. 2024SMILES-Mamba: Chemical Mamba Foundation Models for Drug ADMET Predictioncited by 13
  31. 2025Hybrid Architectures for Language Models: Systematic Analysis and Design Insightscited by 13
  32. 2025Tiled Flash Linear Attention: More Efficient Linear RNN and xLSTM Kernelscited by 12
  33. 2025COSMO: Combination of Selective Memorization for Low-cost Vision-and-Language Navigationcited by 12
  34. 2025Blending Complementary Memory Systems in Hybrid Quadratic-Linear Transformerscited by 12
  35. 2025Scaling Linear Attention with Sparse State Expansioncited by 12
  36. 2024Expansion Span: Combining Fading Memory and Retrieval in Hybrid State Space Modelscited by 11
  37. 2025M4V: Multimodal Mamba for Efficient Text-to-Video Generationcited by 11
  38. 2026Reverso: Efficient Time Series Foundation Models for Zero-shot Forecastingcited by 11

50 most cited of 177 All 177 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts
  2. 2024Revealing and Mitigating the Local Pattern Shortcuts of Mamba
  3. 2024Bi-Mamba: Towards Accurate 1-Bit State Space Models
  4. 2024How Effective are State Space Models for Machine Translation?
  5. 2024BlackMamba: Mixture of Experts for State-Space Models
  6. 2024Venturing into Uncharted Waters: The Navigation Compass from Transformer to Mamba
  7. 2026Interpreting and Steering State-Space Models via Activation Subspace Bottlenecks
  8. 2024The Hidden Attention of Mamba Models