mlsanityData to 7 Oct 2026

Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling

arXiv 2406.07522 ↗First posted 11 Jun 2024cs.CL cs.LGcited by 167

On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language

Abstract (arXiv)

Efficiently modeling sequences with infinite context length has long been a challenging problem. Previous approaches have either suffered from quadratic computational complexity or limited extrapolation ability in length generalization. In this work, we present Samba, a simple hybrid architecture that layer-wise combines Mamba, a selective State Space Model (SSM), with Sliding Window Attention (SWA). Samba selectively compresses a given sequence into recurrent hidden states while still maintaining the ability to precisely recall recent memories with the attention mechanism. We scale Samba up to 3.8B parameters with 3.2T training tokens and demonstrate that it significantly outperforms state-of-the-art models across a variety of benchmarks. Pretrained on sequences of 4K length, Samba shows improved perplexity in context lengths of up to 1M in zero-shot. When finetuned on 4K-length sequences, Samba efficiently extrapolates to a 256K context length with perfect memory recall on the Passkey Retrieval task, and exhibits superior retrieval extrapolation on the challenging Phonebook task compared to full-attention models. As a linear-time sequence model, Samba achieves a 3.73x higher throughput compared to Transformers with grouped-query attention for user prompts of 128K length, and a 3.64x speedup when generating 64K tokens with unlimited streaming. Our code for training on open source data is publicly available at https://github.com/microsoft/Samba.

Key facts. Cited by 167 papers in the map (data through 7 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 89 of its 102 references that are in the map. First posted on arXiv 11 Jun 2024.

Builds on

89 of 102 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Jamba: A Hybrid Transformer-Mamba Language Model28 Mar
  2. 2024Simple linear attention language models balance the recall-throughput tradeoff28 Feb
  3. 2024Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Tasks6 Feb
  4. 2023Gated Linear Attention Transformers with Hardware-Efficient Training11 Dec
  5. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spaces1 Dec
  6. 2022Simplified State Space Layers for Sequence Modeling9 Aug
  7. 2021Efficiently Modeling Long Sequences with Structured State Spaces31 Oct

Other references · 82

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  8. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  10. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  11. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  12. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
38 more
  1. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  2. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  4. 2021Program Synthesis with Large Language Modelscited by 3,509
  5. 2020Longformer: The Long-Document Transformercited by 3,362
  6. 2023Mistral 7Bcited by 3,065
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  8. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  9. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  10. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  11. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  12. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  13. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  14. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  15. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  16. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  17. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  18. 2020GLU Variants Improve Transformercited by 1,724
  19. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  20. 2020Reformer: The Efficient Transformercited by 1,658
  21. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  22. 2019Root Mean Square Layer Normalizationcited by 1,425
  23. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  24. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,085
  25. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  26. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  27. 2020On Layer Normalization in the Transformer Architecturecited by 906
  28. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 721
  29. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  30. 2016Language Modeling with Gated Convolutional Networkscited by 688
  31. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  32. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  33. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  34. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  35. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  36. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  37. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 418
  38. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397

50 most cited of 82 All 82 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 27; 2025: 71; 2026: 69.

167 papers.

Sort:

By area:

  1. 2024Gated Delta Networks: Improving Mamba2 with Delta Rulecited by 483
  2. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 375
  3. 2024Titans: Learning to Memorize at Test Timecited by 303
  4. 2025MiniMax-M1: Scaling Test-Time Compute Efficiently with Lightning Attentioncited by 248
  5. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 120
  6. 2024Hymba: A Hybrid-head Architecture for Small Language Modelscited by 95
  7. 2024Gated Slot Attention for Efficient Linear-Time Sequence Modelingcited by 73
  8. 2025Nested Learning: The Illusion of Deep Learning Architecturescited by 69
  9. 2024Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Modelscited by 66
  10. 2025It's All Connected: A Journey Through Test-Time Memorization, Attentional Bias, Retention, and Online Optimizationcited by 60
  11. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methodscited by 59
  12. 2025Long-Context State-Space Video World Modelscited by 59
38 more
  1. 2024Longhorn: State Space Models are Amortized Online Learnerscited by 58
  2. 2025ATLAS: Learning to Optimally Memorize the Context at Test Timecited by 57
  3. 2025Forgetting Transformer: Softmax Attention with a Forget Gatecited by 50
  4. 2024Falcon Mamba: The First Competitive Attention-free 7B Language Modelcited by 43
  5. 2024LinGen: Towards High-Resolution Minute-Length Text-to-Video Generation with Linear Computational Complexitycited by 41
  6. 2024A Survey of Mambacited by 39
  7. 2025Jet-Nemotron: Efficient Language Model with Post Neural Architecture Searchcited by 39
  8. 2025A Systematic Analysis of Hybrid Linear Attentioncited by 38
  9. 2025Physics of Language Models: Part 4.1, Architecture Design and the Magic of Canon Layerscited by 32
  10. 2024Marconi: Prefix Caching for the Era of Hybrid LLMscited by 29
  11. 2025Next-Latent Prediction Transformers Learn Compact World Modelscited by 29
  12. 2026Gated DeltaNet-2: Decoupling Erase and Write in Linear Attentioncited by 24
  13. 2025Thinking Slow, Fast: Scaling Inference Compute with Distilled Reasonerscited by 22
  14. 2025LongRoPE2: Near-Lossless LLM Context Window Scalingcited by 21
  15. 2025M1: Towards Scalable Test-Time Compute with Mamba Reasoning Modelscited by 21
  16. 2025Understanding and Improving Length Generalization in Recurrent Modelscited by 19
  17. 2025Zebra-Llama: Towards Extremely Efficient Hybrid Modelscited by 18
  18. 2024GoldFinch: High Performance RWKV/Transformer Hybrid with Linear Pre-Fill and Extreme KV-Cache Compressioncited by 17
  19. 2025xLSTM 7B: A Recurrent LLM for Fast and Efficient Inferencecited by 17
  20. 2025Distilling to Hybrid Attention Models via KL-Guided Layer Selectioncited by 17
  21. 2025Reasoning Beyond Limits: Advances and Open Problems for LLMscited by 16
  22. 2025Understanding the Skill Gap in Recurrent Language Models: The Role of the Gather-and-Aggregate Mechanismcited by 15
  23. 2025Structured Sparse Transition Matrices to Enable State Tracking in State-Space Modelscited by 15
  24. 2025Nemotron-Flash: Towards Latency-Optimal Hybrid Small Language Modelscited by 15
  25. 2025iFlame: Interleaving Full and Linear Attention for Efficient Mesh Generationcited by 13
  26. 2025From 128K to 4M: Efficient Training of Ultra-Long Context Large Language Modelscited by 13
  27. 2025Hybrid Architectures for Language Models: Systematic Analysis and Design Insightscited by 13
  28. 2024ReMamba: Equip Mamba with Effective Long-Sequence Modelingcited by 12
  29. 2025Blending Complementary Memory Systems in Hybrid Quadratic-Linear Transformerscited by 12
  30. 2025Comba: Improving Bilinear RNNs with Closed-loop Controlcited by 12
  31. 2026Memory Caching: RNNs with Growing Memorycited by 12
  32. 2025Short window attention enables long-term memorizationcited by 11
  33. 2026Fast-weight Product Key Memorycited by 11
  34. 2025LoLA: Low-Rank Linear Attention With Sparse Cachingcited by 10
  35. 2025Native Hybrid Attention for Efficient Sequence Modelingcited by 10
  36. 2024A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contextscited by 9
  37. 2025Advancing Intelligent Sequence Modeling: Evolution, Trade-offs, and Applications of State-Space Architectures from S4 to Mambacited by 9
  38. 2025Artificial Hippocampus Networks for Efficient Long-Context Modelingcited by 9

50 most cited of 167 All 167 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Taipan: Efficient and Expressive State Space Language Models with Selective Attention
  2. 2024Expansion Span: Combining Fading Memory and Retrieval in Hybrid State Space Models
  3. 2025LongMamba: Enhancing Mamba's Long Context Capabilities via Training-Free Receptive Field Enlargement
  4. 2025Decoder-Hybrid-Decoder Architecture for Efficient Reasoning with Long Generation
  5. 2024Revealing and Mitigating the Local Pattern Shortcuts of Mamba
  6. 2026Retrieval-Aware Distillation for Transformer-SSM Hybrids
  7. 2025MemMamba: Rethinking Memory Patterns in State Space Model
  8. 2026Rethinking the Role of Efficient Attention in Hybrid Architectures