mlsanityData to 7 Oct 2026

Jamba-1.5: Hybrid Transformer-Mamba Models at Scale

arXiv 2408.12570 ↗First posted 22 Aug 2024cs.CL cs.LGcited by 48

On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language

Abstract (arXiv)

We present Jamba-1.5, new instruction-tuned large language models based on our Jamba architecture. Jamba is a hybrid Transformer-Mamba mixture of experts architecture, providing high throughput and low memory usage across context lengths, while retaining the same or better quality as Transformer models. We release two model sizes: Jamba-1.5-Large, with 94B active parameters, and Jamba-1.5-Mini, with 12B active parameters. Both models are fine-tuned for a variety of conversational and instruction-following capabilties, and have an effective context length of 256K tokens, the largest amongst open-weight models. To support cost-effective inference, we introduce ExpertsInt8, a novel quantization technique that allows fitting Jamba-1.5-Large on a machine with 8 80GB GPUs when processing 256K-token contexts without loss of quality. When evaluated on a battery of academic and chatbot benchmarks, Jamba-1.5 models achieve excellent results while providing high throughput and outperforming other open-weight models on long-context benchmarks. The model weights for both sizes are publicly available under the Jamba Open Model License and we release ExpertsInt8 as open source.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 32 of its 41 references that are in the map. First posted on arXiv 22 Aug 2024.

Builds on

32 of 41 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024An Empirical Study of Mamba-based Language Models12 Jun
  2. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality31 May
  3. 2024Jamba: A Hybrid Transformer-Mamba Language Model28 Mar
  4. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spaces1 Dec
  5. 2021Efficiently Modeling Long Sequences with Structured State Spaces31 Oct

Other references · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  8. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  11. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  12. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
15 more
  1. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  2. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  3. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  4. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,767
  5. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  6. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  7. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  8. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  9. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  10. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
  11. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 211
  12. 2022MegaBlocks: Efficient Sparse Training with Mixture-of-Expertscited by 204
  13. 2023Okapi: Instruction-tuned Large Language Models in Multiple Languages with Reinforcement Learning from Human Feedbackcited by 189
  14. 2023Zoology: Measuring and Improving Recall in Efficient Language Modelscited by 188
  15. 2024The 2024 Foundation Model Transparency Indexcited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 27; 2026: 6.

48 papers.

Sort:

By area:

  1. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  2. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 200
  3. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  4. 2025NoLiMa: Long-Context Evaluation Beyond Literal Matchingcited by 120
  5. 2025A Comprehensive Survey of Mixture-of-Experts: Algorithms, Theory, and Applicationscited by 91
  6. 2024Hunyuan-Large: An Open-Source MoE Model with 52 Billion Activated Parameters by Tencentcited by 89
  7. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 74
  8. 2025We're Different, We're the Same: Creative Homogeneity Across LLMscited by 41
  9. 2025A Systematic Analysis of Hybrid Linear Attentioncited by 38
  10. 2024Marconi: Prefix Caching for the Era of Hybrid LLMscited by 29
  11. 2024The Zamba2 Suite: Technical Reportcited by 20
  12. 2024GAMEBoT: Transparent Assessment of LLM Reasoning in Gamescited by 16
36 more
  1. 2025GSM-Infinite: How Do Your LLMs Behave over Infinitely Increasing Context Length and Reasoning Complexity?cited by 16
  2. 2025Quamba2: A Robust and Scalable Post-training Quantization Framework for Selective State Space Modelscited by 14
  3. 2025Hybrid Architectures for Language Models: Systematic Analysis and Design Insightscited by 13
  4. 2024Needle Threading: Can LLMs Follow Threads through Near-Million-Scale Haystacks?cited by 12
  5. 2024Contextualized Evaluations: Judging Language Model Responses to Underspecified Queriescited by 12
  6. 2025Systems and Algorithms for Convolutional Multi-Hybrid Language Models at Scalecited by 12
  7. 2025The Markovian Thinker: Architecture-Agnostic Linear Scaling of Reasoningcited by 12
  8. 2024LightTransfer: Your Long-Context LLM is Secretly a Hybrid Model with Effortless Adaptationcited by 11
  9. 2024On Many-Shot In-Context Learning for Long-Context Evaluationcited by 10
  10. 2024State Space Models are Strong Text Rerankerscited by 10
  11. 2025From Markov to Laplace: How Mamba In-Context Learns Markov Chainscited by 9
  12. 2025Advancing Intelligent Sequence Modeling: Evolution, Trade-offs, and Applications of State-Space Architectures from S4 to Mambacited by 9
  13. 2025TransMamba: Fast Universal Architecture Adaption from Transformers to Mambacited by 6
  14. 2025Linear-MoE: Linear Sequence Modeling Meets Mixture-of-Expertscited by 6
  15. 20247B Fully Open Source Moxin-LLM/VLM -- From Pretraining to GRPO-based Reinforcement Learning Enhancementcited by 5
  16. 2025CryptoX : Compositional Reasoning Evaluation of Large Language Modelscited by 5
  17. 2025Overflow Prevention Enhances Long-Context Recurrent LLMscited by 5
  18. 2025What happens when generative AI models train recursively on each others' outputs?cited by 5
  19. 2025Open-Source Multimodal Moxin Models with Moxin-VLM and Moxin-VLAcited by 4
  20. 2026Kwai Summary Attention Technical Reportcited by 4
  21. 2024GG-SSMs: Graph-Generating State Space Modelscited by 3
  22. 2025GAMDTP: Dynamic Trajectory Prediction with Graph Attention Mamba Networkcited by 2
  23. 2025Nirvana: A Specialized Generalist Model With Task-Aware Memory Mechanismcited by 2
  24. 2025Apriel-H1: Towards Efficient Enterprise Reasoning Modelscited by 2
  25. 2026Hidden State Poisoning Attacks against Mamba-based Language Modelscited by 2
  26. 2025Causes and Consequences of Representational Similarity in Machine Learning Modelscited by 1
  27. 2025Scaling Legal AI: Benchmarking Mamba and Transformers for Statutory Classification and Case Law Retrievalcited by 1
  28. 2025UAM: A Unified Attention-Mamba Backbone of Multimodal Framework for Tumor Cell Classificationcited by 1
  29. 2026Retrievit: In-context Retrieval Capabilities of Transformers, State Space Models, and Hybrid Architecturescited by 1
  30. 2026Taylor-Calibrate: Principled Initialization for Hybrid Linear Attention Distillationcited by 1
  31. 2024How much do contextualized representations encode long-range context?cited by 0
  32. 2025From Divergence to Consensus: Evaluating the Role of Large Language Models in Facilitating Agreement through Adaptive Strategiescited by 0
  33. 2025Balancing Computation Load and Representation Expressivity in Parallel Hybrid Neural Networkscited by 0
  34. 2025ParallelTime: Dynamically Weighting the Balance of Short- and Long-Term Temporal Dependenciescited by 0
  35. 2026NOVA: Technology-Architecture Co-Design of Near-Memory Processing for Attention-SSM-MoE Hybrid LLM Inferencecited by 0
  36. 2026Mechanics of Long-Context Hybrid Models Part 1.1: From Hybrid Attention to Hybrid Positioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Models
  2. 2024Bi-Mamba: Towards Accurate 1-Bit State Space Models
  3. 2025Llamba: Scaling Distilled Recurrent Models for Efficient Language Processing
  4. 2024BlackMamba: Mixture of Experts for State-Space Models
  5. 2025Routing Mamba: Scaling State Space Models with Mixture-of-Experts Projection
  6. 2024PackMamba: Efficient Processing of Variable-Length Sequences in Mamba training
  7. 2024MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts
  8. 2025TransMamba: A Sequence-Level Hybrid Transformer-Mamba Language Model