mlsanityData to 14 Oct 2026

The Zamba2 Suite: Technical Report

arXiv 2411.15242 ↗First posted 22 Nov 2024cs.LG cs.AI cs.CLcited by 21

Not on the map. Why?

Abstract (arXiv)

In this technical report, we present the Zamba2 series -- a suite of 1.2B, 2.7B, and 7.4B parameter hybrid Mamba2-transformer models that achieve state of the art performance against the leading open-weights models of their class, while achieving substantial gains in inference latency, throughput, and memory efficiency. The Zamba2 series builds upon our initial work with Zamba1-7B, optimizing its architecture, training and annealing datasets, and training for up to three trillion tokens. We provide open-source weights for all models of the Zamba2 series as well as instruction-tuned variants that are strongly competitive against comparable instruct-tuned models of their class. We additionally open-source the pretraining dataset, which we call Zyda-2, used to train the Zamba2 series of models. The models and datasets used in this work are openly available at https://huggingface.co/Zyphra

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 39 of its 59 references that are in the map. First posted on arXiv 22 Nov 2024.

Builds on

39 of 59 references.

References · 39

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  8. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  10. 2023Mistral 7Bcited by 3,164
  11. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,367
  12. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
27 more
  1. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  2. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  3. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  4. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,228
  5. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  6. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  7. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  8. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  9. 2023StarCoder: may the source be with you!cited by 799
  10. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 723
  11. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
  12. 2017Deep Learning Scaling is Predictable, Empiricallycited by 582
  13. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  14. 2023Gated Linear Attention Transformers with Hardware-Efficient Trainingcited by 514
  15. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 483
  16. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  17. 2024DataComp-LM: In search of the next generation of training sets for language modelscited by 398
  18. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
  19. 2024An Empirical Study of Mamba-based Language Modelscited by 186
  20. 2024Simple and Scalable Strategies to Continually Pre-train Large Language Modelscited by 127
  21. 2024Zamba: A Compact 7B SSM Hybrid Modelcited by 124
  22. 2024Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Taskscited by 105
  23. 2024Does your data spark joy? Performance gains from domain upsampling at the end of trainingcited by 52
  24. 2024Jamba-1.5: Hybrid Transformer-Mamba Models at Scalecited by 50
  25. 2024Falcon Mamba: The First Competitive Attention-free 7B Language Modelcited by 46
  26. 2024BlackMamba: Mixture of Experts for State-Space Modelscited by 37
  27. 2024The Case for Co-Designing Model Architectures with Hardwarecited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 8; 2026: 12.

21 papers.

Sort:

By area:

  1. 2025Don't be lazy: CompleteP enables compute-efficient deep transformerscited by 78
  2. 2025LongMamba: Enhancing Mamba's Long Context Capabilities via Training-Free Receptive Field Enlargementcited by 42
  3. 2025Jet-Nemotron: Efficient Language Model with Post Neural Architecture Searchcited by 41
  4. 2025Hybrid Architectures for Language Models: Systematic Analysis and Design Insightscited by 14
  5. 2026Falcon-H1R: Pushing the Reasoning Frontiers with a Hybrid Model for Efficient Test-Time Scalingcited by 13
  6. 2024Expansion Span: Combining Fading Memory and Retrieval in Hybrid State Space Modelscited by 11
  7. 2025Scaling with Collapse: Efficient and Predictable Training of LLM Familiescited by 9
  8. 2025SparseSSM: Efficient Selective Structured State Space Models Can Be Pruned in One-Shotcited by 5
  9. 2025Compressed Convolutional Attention: Efficient Attention in a Compressed Latent Spacecited by 5
  10. 2026The Capability Convergence Hypothesis: Capability from Access Structure, Not Scalecited by 4
  11. 2026When to Think Fast and Slow? AMOR: Adaptive Entropy Gate for Hybrid Modelscited by 3
  12. 2026Massive Activations in Hybrid Linear Attention Large Language Models: Pre-Attention Spikes and Inter-Spike Plateauscited by 2
9 more
  1. 2025Characterizing State Space Model and Hybrid Language Model Performance with Long Contextcited by 1
  2. 2026Online Vector Quantized Attentioncited by 1
  3. 2026Context Cartography: Toward Structured Governance of Contextual Space in Large Language Model Systemscited by 1
  4. 2026Hybrid Associative Memoriescited by 1
  5. 2026The Impossibility Triangle of Long-Context Modelingcited by 1
  6. 2026LayerBoost: Layer-Aware Attention Reduction for Efficient LLMscited by 0
  7. 2026Zamba2-VL Technical Reportcited by 0
  8. 2026Forget Attention: Importance-Aware Attention Is All You Needcited by 0
  9. 2026The Evolution of Attention in Large Language Models: Mechanisms, Trade-offs, and Emerging Trendscited by 0