mlsanityData to 14 Oct 2026

Jamba: A Hybrid Transformer-Mamba Language Model

arXiv 2403.19887 ↗First posted 28 Mar 2024cs.CL cs.LGcited by 401

On the map: Mixture-of-experts → MoE large language models

also in 1 more
Abstract (arXiv)

We present Jamba, a new base large language model based on a novel hybrid Transformer-Mamba mixture-of-experts (MoE) architecture. Specifically, Jamba interleaves blocks of Transformer and Mamba layers, enjoying the benefits of both model families. MoE is added in some of these layers to increase model capacity while keeping active parameter usage manageable. This flexible architecture allows resource- and objective-specific configurations. In the particular configuration we have implemented, we end up with a powerful model that fits in a single 80GB GPU. Built at large scale, Jamba provides high throughput and small memory footprint compared to vanilla Transformers, and at the same time state-of-the-art performance on standard language model benchmarks and long-context evaluations. Remarkably, the model presents strong results for up to 256K tokens context length. We study various architectural decisions, such as how to combine Transformer and Mamba layers, and how to mix experts, and show that some of them are crucial in large scale modeling. We also describe several interesting properties of these architectures which the training and evaluation of Jamba have revealed, and plan to release checkpoints from various ablation runs, to encourage further exploration of this novel architecture. We make the weights of our implementation of Jamba publicly available under a permissive license.

Key facts. Cited by 401 papers in the map (data through 14 Oct 2026). Placed in MoE large language models, one of 12 directions in Mixture-of-experts. Builds on 46 of its 56 references that are in the map. First posted on arXiv 28 Mar 2024.

Builds on

46 of 56 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Mixtral of Experts8 Jan
  2. 2024MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts8 Jan
  3. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Models17 Feb
  4. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer23 Jan

Other references · 42

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spacescited by 4,718
  9. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  10. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  11. 2023Mistral 7Bcited by 3,164
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
30 more
  1. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  2. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  3. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  4. 2021Efficiently Modeling Long Sequences with Structured State Spacescited by 2,465
  5. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  6. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  7. 2020GLU Variants Improve Transformercited by 1,804
  8. 2019Root Mean Square Layer Normalizationcited by 1,480
  9. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  10. 2022In-context Learning and Induction Headscited by 964
  11. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 608
  12. 2017The NarrativeQA Reading Comprehension Challengecited by 597
  13. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 499
  14. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 433
  15. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 393
  16. 2019An Evaluation Dataset for Intent Classification and Out-of-Scope Predictioncited by 386
  17. 2018QuAC : Question Answering in Contextcited by 381
  18. 2022Unified Scaling Laws for Routed Language Modelscited by 253
  19. 2021CUAD: An Expert-Annotated NLP Dataset for Legal Contract Reviewcited by 236
  20. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 233
  21. 2019Benchmarking Natural Language Understanding Services for building Conversational Agentscited by 166
  22. 2022Transformer Language Models without Positional Encodings Still Learn Positional Informationcited by 150
  23. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 132
  24. 2024Can Mamba Learn How to Learn? A Comparative Study on In-Context Learning Taskscited by 105
  25. 2024The Hidden Attention of Mamba Modelscited by 91
  26. 2022Parallel Context Windows for Large Language Modelscited by 81
  27. 2022Efficient Long Sequence Modeling via State Space Augmented Transformercited by 37
  28. 2023Diagonal State Space Augmented Transformers for Speech Recognitioncited by 25
  29. 2023Block-State Transformerscited by 24
  30. 2023Multi-Head State Space Model for Speech Recognitioncited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 134; 2025: 131; 2026: 136.

401 papers.

Sort:

By area:

  1. 2024Qwen2 Technical Reportcited by 2,120
  2. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Dualitycited by 1,336
  3. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 396
  4. 2024MInference 1.0: Accelerating Pre-filling for Long-Context LLMs via Dynamic Sparse Attentioncited by 383
  5. 2025Memory in the Age of AI Agentscited by 303
  6. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 301
  7. 2024A Survey on Mixture of Experts in Large Language Modelscited by 237
  8. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 215
  9. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 208
  10. 2025Kimi Linear: An Expressive, Efficient Attention Architecturecited by 196
  11. 2024An Empirical Study of Mamba-based Language Modelscited by 186
  12. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modelingcited by 174
38 more
  1. 2024How to Train Long-Context Language Models (Effectively)cited by 146
  2. 2024You Only Cache Once: Decoder-Decoder Architectures for Language Modelscited by 144
  3. 2024MambaOut: Do We Really Need Mamba for Vision?cited by 143
  4. 2024Zamba: A Compact 7B SSM Hybrid Modelcited by 124
  5. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 123
  6. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 122
  7. 2024Long-LRM: Long-sequence Large Reconstruction Model for Wide-coverage Gaussian Splatscited by 114
  8. 2024Demystify Mamba in Vision: A Linear Attention Perspectivecited by 102
  9. 2024LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecturecited by 102
  10. 2025Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Modelscited by 100
  11. 2024Hymba: A Hybrid-head Architecture for Small Language Modelscited by 99
  12. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  13. 2024DRAMA: An Efficient End-to-end Motion Planner for Autonomous Driving with Mambacited by 87
  14. 2024LLMs Will Always Hallucinate, and We Need to Live With Thiscited by 78
  15. 2024Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Modelscited by 67
  16. 2024Mamba-360: Survey of State Space Models as Transformer Alternative for Long Sequence Modelling: Methods, Applications, and Challengescited by 64
  17. 2024Why Does the Effective Context Length of LLMs Fall Short?cited by 63
  18. 2024Mamba in Speech: Towards an Alternative to Self-Attentioncited by 60
  19. 2024SCBench: A KV Cache-Centric Analysis of Long-Context Methodscited by 60
  20. 2024State Space Model for New-Generation Network Alternative to Transformers: A Surveycited by 59
  21. 2024Compression Represents Intelligence Linearlycited by 56
  22. 2024A Survey on Visual Mambacited by 50
  23. 2024Jamba-1.5: Hybrid Transformer-Mamba Models at Scalecited by 50
  24. 2024DeciMamba: Exploring the Length Extrapolation Potential of Mambacited by 49
  25. 2024A Survey on Inference Optimization Techniques for Mixture of Experts Modelscited by 47
  26. 2025Massive Values in Self-Attention Modules are the Key to Contextual Knowledge Understandingcited by 47
  27. 2024Fishing for Magikarp: Automatically Detecting Under-trained Tokens in Large Language Modelscited by 46
  28. 2024Falcon Mamba: The First Competitive Attention-free 7B Language Modelcited by 46
  29. 2024Hamba: Single-view 3D Hand Reconstruction with Graph-guided Bi-Scanning Mambacited by 45
  30. 2024KV Cache Compression, But What Must We Give in Return? A Comprehensive Benchmark of Long Context Capable Approachescited by 42
  31. 2024Mamba-R: Vision Mamba ALSO Needs Registerscited by 41
  32. 2025Jet-Nemotron: Efficient Language Model with Post Neural Architecture Searchcited by 41
  33. 2024A Survey of Mambacited by 39
  34. 2024Context Clues: Evaluating Long Context Models for Clinical Prediction Tasks on EHRscited by 39
  35. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspectivecited by 38
  36. 2025Vamba: Understanding Hour-Long Videos with Hybrid Mamba-Transformerscited by 36
  37. 2024Towards Neural Scaling Laws for Time Series Foundation Modelscited by 33
  38. 2025Liger: Linearizing Large Language Models to Gated Recurrent Structurescited by 33

50 most cited of 401 All 401 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024PackMamba: Efficient Processing of Variable-Length Sequences in Mamba training
  2. 2025Pimba: A Processing-in-Memory Acceleration for Post-Transformer Large Language Model Serving
  3. 2025TransMamba: Fast Universal Architecture Adaption from Transformers to Mamba
  4. 2025Multimodal Mamba: Decoder-only Multimodal State Space Model via Quadratic to Linear Distillation
  5. 2025Llamba: Scaling Distilled Recurrent Models for Efficient Language Processing
  6. 2024BlackMamba: Mixture of Experts for State-Space Models
  7. 2024Bi-Mamba: Towards Accurate 1-Bit State Space Models
  8. 2025DYNAMAX: Dynamic computing for Transformers and Mamba based architectures