mlsanityData to 7 Oct 2026

Zamba: A Compact 7B SSM Hybrid Model

arXiv 2405.16712 ↗First posted 26 May 2024cs.LG cs.AI cs.CLcited by 118

Not on the map. Why?

Cited by 29 in State-space sequence models (S4, Mamba), but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In this technical report, we present Zamba, a novel 7B SSM-transformer hybrid model which achieves competitive performance against leading open-weight models at a comparable scale. Zamba is trained on 1T tokens from openly available datasets and is the best non-transformer model at this scale. Zamba pioneers a unique architecture combining a Mamba backbone with a single shared attention module, thus obtaining the benefits of attention at minimal parameter cost. Due to its architecture, Zamba is significantly faster at inference than comparable transformer models and requires substantially less memory for generation of long sequences. Zamba is pretrained in two phases: the first phase is based on existing web datasets, while the second one consists of annealing the model over high-quality instruct and synthetic datasets, and is characterized by a rapid learning rate decay. We open-source the weights and all checkpoints for Zamba, through both phase 1 and annealing phases.

Key facts. Cited by 118 papers in the map (data through 7 Oct 2026). Builds on 48 of its 62 references that are in the map. First posted on arXiv 26 May 2024.

Builds on

48 of 62 references.

References · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  9. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  10. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  11. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  12. 2022Training Compute-Optimal Large Language Modelscited by 2,927
36 more
  1. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  2. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,424
  3. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  4. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  5. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  6. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  7. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  8. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  9. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  10. 2018Universal Transformerscited by 613
  11. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
  12. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 600
  13. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  14. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  15. 2023Textbooks Are All You Needcited by 561
  16. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  17. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
  18. 2024Jamba: A Hybrid Transformer-Mamba Language Modelcited by 390
  19. 2023DoReMi: Optimizing Data Mixtures Speeds Up Language Model Pretrainingcited by 385
  20. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 232
  21. 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 208
  22. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 156
  23. 2023D4: Improving LLM Pretraining via Document De-Duplication and Diversificationcited by 146
  24. 2023Continual Pre-Training of Large Language Models: How to (re)warm your model?cited by 135
  25. 2024Simple and Scalable Strategies to Continually Pre-train Large Language Modelscited by 122
  26. 2024Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Taskscited by 104
  27. 2023LLM360: Towards Fully Transparent Open-Source LLMscited by 100
  28. 2021Relating transformers to models and neural representations of the hippocampal formationcited by 62
  29. 2024Mechanistic Design and Scaling of Hybrid Architecturescited by 57
  30. 2024JetMoE: Reaching Llama2 Performance with 0.1M Dollarscited by 55
  31. 2024Is Mamba Capable of In-Context Learning?cited by 45
  32. 2023Scaling MLPs: A Tale of Inductive Biascited by 37
  33. 2024BlackMamba: Mixture of Experts for State-Space Modelscited by 37
  34. 2022Hybrid Predictive Coding: Inferring, Fast and Slowcited by 25
  35. 2022Downstream Datasets Make Surprisingly Good Pretraining Corporacited by 20
  36. 2020Going in circles is the way forward: the role of recurrence in visual inferencecited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 28; 2025: 49; 2026: 41.

118 papers.

Sort:

By area:

  1. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,283
  2. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  3. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  4. 2024An Empirical Study of Mamba-based Language Modelscited by 177
  5. 2024Hymba: A Hybrid-head Architecture for Small Language Modelscited by 95
  6. 2024Relaxed Recursive Transformers: Effective Parameter Sharing with Layer-wise LoRAcited by 72
  7. 2025Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distributioncited by 70
  8. 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 69
  9. 2024Falcon Mamba: The First Competitive Attention-free 7B Language Modelcited by 43
  10. 2025A Systematic Analysis of Hybrid Linear Attentioncited by 38
  11. 2025LongMamba: Enhancing Mamba's Long Context Capabilities via Training-Free Receptive Field Enlargementcited by 36
  12. 2025Llamba: Scaling Distilled Recurrent Models for Efficient Language Processingcited by 35
38 more
  1. 2025Vamba: Understanding Hour-Long Videos with Hybrid Mamba-Transformerscited by 35
  2. 2025Liger: Linearizing Large Language Models to Gated Recurrent Structurescited by 32
  3. 2024Marconi: Prefix Caching for the Era of Hybrid LLMscited by 29
  4. 2024LoLCATs: On Low-Rank Linearizing of Large Language Modelscited by 25
  5. 2024Maximize Your Data's Potential: Enhancing LLM Accuracy with Two-Phase Pretrainingcited by 24
  6. 2024The Zamba2 Suite: Technical Reportcited by 20
  7. 2024SLIM: Let LLM Learn More and Forget Less with Soft LoRA and Identity Mixturecited by 19
  8. 2024Long-Form Speech Generation with Spoken Language Modelscited by 19
  9. 2025Mid-Training of Large Language Models: A Surveycited by 18
  10. 2025xLSTM 7B: A Recurrent LLM for Fast and Efficient Inferencecited by 17
  11. 2025Understanding the Skill Gap in Recurrent Language Models: The Role of the Gather-and-Aggregate Mechanismcited by 15
  12. 2025Nemotron-Flash: Towards Latency-Optimal Hybrid Small Language Modelscited by 15
  13. 2024Tree Attention: Topology-aware Decoding for Long-Context Attention on GPU clusterscited by 13
  14. 2025Hybrid Architectures for Language Models: Systematic Analysis and Design Insightscited by 13
  15. 2024Zyda-2: a 5 Trillion Token High-Quality Datasetcited by 12
  16. 2025Systems and Algorithms for Convolutional Multi-Hybrid Language Models at Scalecited by 12
  17. 2025Scaling Linear Attention with Sparse State Expansioncited by 12
  18. 2024DiMSUM: Diffusion Mamba -- A Scalable and Unified Spatial-Frequency Method for Image Generationcited by 11
  19. 2024Expansion Span: Combining Fading Memory and Retrieval in Hybrid State Space Modelscited by 11
  20. 2026Reverso: Efficient Time Series Foundation Models for Zero-shot Forecastingcited by 11
  21. 2024State Space Models are Strong Text Rerankerscited by 10
  22. 2025LoLA: Low-Rank Linear Attention With Sparse Cachingcited by 10
  23. 2025Native Hybrid Attention for Efficient Sequence Modelingcited by 10
  24. 2024Crosslingual Capabilities and Knowledge Barriers in Multilingual Large Language Modelscited by 9
  25. 2024A Little Goes a Long Way: Efficient Long Context Training and Inference with Partial Contextscited by 9
  26. 2025Advancing Intelligent Sequence Modeling: Evolution, Trade-offs, and Applications of State-Space Architectures from S4 to Mambacited by 9
  27. 2025Gated KalmaNet: A Fading Memory Layer Through Test-Time Ridge Regressioncited by 9
  28. 2025InfiniteVL: Synergizing Linear and Sparse Attention for Highly-Efficient, Unlimited-Input Vision-Language Modelscited by 9
  29. 2025STree: Speculative Tree Decoding for Hybrid State-Space Modelscited by 8
  30. 2025RAD: Redundancy-Aware Distillation for Hybrid Models via Self-Speculative Decodingcited by 8
  31. 2025Mamba-Shedder: Post-Transformer Compression for Efficient Selective Structured State Space Modelscited by 7
  32. 2025Minitron-SSM: Efficient Hybrid Language Model Compression through Group-Aware SSM Pruningcited by 7
  33. 2025HybriDNA: A Hybrid Transformer-Mamba2 Long-Range DNA Language Modelcited by 6
  34. 2025Single-Pass Document Scanning for Question Answeringcited by 6
  35. 2025RWKV-X: A Linear Complexity Hybrid Language Modelcited by 6
  36. 2025Lizard: An Efficient Linearization Framework for Large Language Modelscited by 6
  37. 2025Understanding and Improving Length Generalization in Hierarchical Sparse Attention Modelscited by 6
  38. 2024How Effective are State Space Models for Machine Translation?cited by 5

50 most cited of 118 All 118 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025TransMamba: A Sequence-Level Hybrid Transformer-Mamba Language Model
  2. 2025Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models
  3. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Models
  4. 2024A Survey of Mamba
  5. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spaces
  6. 2026InfoMamba: An Attention-Free Hybrid Mamba-Transformer Model
  7. 2025DYNAMAX: Dynamic computing for Transformers and Mamba based architectures
  8. 2024Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models