mlsanityData to 7 Oct 2026

The Mamba in the Llama: Distilling and Accelerating Hybrid Models

arXiv 2408.15237 ↗First posted 27 Aug 2024cs.LG cs.AIcited by 120

On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language

Abstract (arXiv)

Linear RNN architectures, like Mamba, can be competitive with Transformer models in language modeling while having advantageous deployment characteristics. Given the focus on training large-scale Transformer models, we consider the challenge of converting these pretrained models for deployment. We demonstrate that it is feasible to distill large Transformers into linear RNNs by reusing the linear projection weights from attention layers with academic GPU resources. The resulting hybrid model, which incorporates a quarter of the attention layers, achieves performance comparable to the original Transformer in chat benchmarks and outperforms open-source hybrid Mamba models trained from scratch with trillions of tokens in both chat benchmarks and general benchmarks. Moreover, we introduce a hardware-aware speculative decoding algorithm that accelerates the inference speed of Mamba and hybrid models. Overall we show how, with limited computation resources, we can remove many of the original attention layers and generate from the resulting model more efficiently. Our top-performing model, distilled from Llama3-8B-Instruct, achieves a 29.61 length-controlled win rate on AlpacaEval 2 against GPT-4 and 7.35 on MT-Bench, surpassing the best 8B scale instruction-tuned linear RNN model. We also find that the distilled model has natural length extrapolation, showing almost perfect accuracy in the needle-in-a-haystack test at 20x the distillation length. Code and pre-trained checkpoints are open-sourced at https://github.com/jxiw/MambaInLlama and https://github.com/itsdaniele/speculative_mamba.

Key facts. Cited by 120 papers in the map (data through 7 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 73 of its 85 references that are in the map. First posted on arXiv 27 Aug 2024.

Builds on

73 of 85 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models19 Aug
  2. 2024An Empirical Study of Mamba-based Language Models12 Jun
  3. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling11 Jun
  4. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality31 May
  5. 2024Jamba: A Hybrid Transformer-Mamba Language Model28 Mar
  6. 2024Simple linear attention language models balance the recall-throughput tradeoff28 Feb
  7. 2023Gated Linear Attention Transformers with Hardware-Efficient Training11 Dec
  8. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spaces1 Dec
  9. 2022Diagonal State Spaces are as Effective as Structured State Spaces27 Mar
  10. 2021Efficiently Modeling Long Sequences with Structured State Spaces31 Oct

Other references · 63

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2024The Llama 3 Herd of Modelscited by 14,192
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  9. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  10. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  11. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  12. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
38 more
  1. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  2. 2023Mistral 7Bcited by 3,065
  3. 2016Pointer Sentinel Mixture Modelscited by 2,992
  4. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  6. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  7. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  8. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  9. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  10. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  11. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  12. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  13. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  14. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  15. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  16. 2023StarCoder: may the source be with you!cited by 790
  17. 2016Sequence-Level Knowledge Distillationcited by 761
  18. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  19. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  20. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 584
  21. 2023Zephyr: Direct Distillation of LM Alignmentcited by 498
  22. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  23. 2021Linear Transformers Are Secretly Fast Weight Programmerscited by 471
  24. 2022On the Parameterization and Initialization of Diagonal State Space Modelscited by 418
  25. 2023Hyena Hierarchy: Towards Larger Convolutional Language Modelscited by 388
  26. 2024Parallelizing Linear Transformers with the Delta Rule over Sequence Lengthcited by 373
  27. 2024xLSTM: Extended Long Short-Term Memorycited by 320
  28. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 295
  29. 2024CRUXEval: A Benchmark for Code Reasoning, Understanding and Executioncited by 294
  30. 2022Long Range Language Modeling via Gated State Spacescited by 265
  31. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  32. 2024Griffin: Mixing Gated Linear Recurrences with Local Attention for Efficient Language Modelscited by 232
  33. 2023Zoology: Measuring and Improving Recall in Efficient Language Modelscited by 188
  34. 2022SCROLLS: Standardized CompaRison Over Long Language Sequencescited by 170
  35. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generationcited by 169
  36. 2023Accelerating LLM Inference with Staged Speculative Decodingcited by 151
  37. 2023REST: Retrieval-Based Speculative Decodingcited by 148
  38. 2024The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicrycited by 119

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 61; 2026: 48.

120 papers.

Sort:

By area:

  1. 2024Scaling Diffusion Language Models via Adaptation from Autoregressive Modelscited by 268
  2. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 188
  3. 2024How to Train Long-Context Language Models (Effectively)cited by 141
  4. 2025Test-Time Training Done Rightcited by 121
  5. 2024LongLLaVA: Scaling Multi-modal LLMs to 1000 Images Efficiently via a Hybrid Architecturecited by 100
  6. 2024Gated Slot Attention for Efficient Linear-Time Sequence Modelingcited by 73
  7. 2025End-to-End Test-Time Training for Long Contextcited by 53
  8. 2025STORM: Token-Efficient Long Video Understanding for Multimodal LLMscited by 46
  9. 2025Jet-Nemotron: Efficient Language Model with Post Neural Architecture Searchcited by 39
  10. 2025Position: Episodic Memory is the Missing Piece for Long-Term LLM Agentscited by 35
  11. 2025Llamba: Scaling Distilled Recurrent Models for Efficient Language Processingcited by 35
  12. 2024LoLCATs: On Low-Rank Linearizing of Large Language Modelscited by 25
38 more
  1. 2025RADLADS: Rapid Attention Distillation to Linear Attention Decoders at Scalecited by 24
  2. 2025Speculative Decoding and Beyond: An In-Depth Survey of Techniquescited by 22
  3. 2025Thinking Slow, Fast: Scaling Inference Compute with Distilled Reasonerscited by 22
  4. 2025M1: Towards Scalable Test-Time Compute with Mamba Reasoning Modelscited by 21
  5. 2025Zebra-Llama: Towards Extremely Efficient Hybrid Modelscited by 18
  6. 2025Distilling to Hybrid Attention Models via KL-Guided Layer Selectioncited by 17
  7. 2026Hybrid Linear Attention Done Right: Efficient Distillation and Effective Architectures for Extremely Long Contextscited by 16
  8. 2026VGG-T3: Offline Feed-Forward 3D Reconstruction at Scalecited by 14
  9. 2025Mamba Modulation: On the Length Generalization of Mambacited by 13
  10. 2025Attention Surgery: An Efficient Recipe to Linearize Your Video Diffusion Transformercited by 13
  11. 2025Hybrid Architectures for Language Models: Systematic Analysis and Design Insightscited by 13
  12. 2026Speculative Speculative Decodingcited by 13
  13. 2025Exploring Diffusion Transformer Designs via Graftingcited by 12
  14. 2024LightTransfer: Your Long-Context LLM is Secretly a Hybrid Model with Effortless Adaptationcited by 11
  15. 2025Multimodal Mamba: Decoder-only Multimodal State Space Model via Quadratic to Linear Distillationcited by 11
  16. 2025M4V: Multimodal Mamba for Efficient Text-to-Video Generationcited by 11
  17. 2024State Space Models are Strong Text Rerankerscited by 10
  18. 2025LoLA: Low-Rank Linear Attention With Sparse Cachingcited by 10
  19. 2025Cache Me If You Can: How Many KVs Do You Need for Effective Long-Context LMs?cited by 10
  20. 2025Native Hybrid Attention for Efficient Sequence Modelingcited by 10
  21. 2025ARWKV: Pretrain is not what we need, an RNN-Attention-Based Language Model Born from Transformercited by 9
  22. 2025Artificial Hippocampus Networks for Efficient Long-Context Modelingcited by 9
  23. 2024Autocorrelation Matters: Understanding the Role of Initialization Schemes for State Space Modelscited by 8
  24. 2025STree: Speculative Tree Decoding for Hybrid State-Space Modelscited by 8
  25. 2025RAD: Redundancy-Aware Distillation for Hybrid Models via Self-Speculative Decodingcited by 8
  26. 2025DeepVIS: Bridging Natural Language and Data Visualization Through Step-wise Reasoningcited by 8
  27. 2025TransMamba: A Sequence-Level Hybrid Transformer-Mamba Language Modelcited by 7
  28. 2025TransMamba: Fast Universal Architecture Adaption from Transformers to Mambacited by 6
  29. 2025On Pruning State-Space LLMscited by 6
  30. 2025MaTVLM: Hybrid Mamba-Transformer for Efficient Vision-Language Modelingcited by 6
  31. 2025Lizard: An Efficient Linearization Framework for Large Language Modelscited by 6
  32. 2025Understanding and Improving Length Generalization in Hierarchical Sparse Attention Modelscited by 6
  33. 2026ReHyAt: Recurrent Hybrid Attention for Video Diffusion Transformerscited by 6
  34. 2025ViT-Linearizer: Distilling Quadratic Knowledge into Linear-Time Vision Modelscited by 5
  35. 2025SpikingBrain: Spiking Brain-inspired Large Modelscited by 5
  36. 2025SpecMamba: Accelerating Mamba Inference on FPGA with Speculative Decodingcited by 5
  37. 2026Retrieval-Aware Distillation for Transformer-SSM Hybridscited by 5
  38. 2026CARE: Covariance-Aware and Rank-Enhanced Decomposition for Enabling Multi-Head Latent Attentioncited by 5

50 most cited of 120 All 120 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Jamba-1.5: Hybrid Transformer-Mamba Models at Scale
  2. 2025Hunyuan-TurboS: Advancing Large Language Models through Mamba-Transformer Synergy and Adaptive Chain-of-Thought
  3. 2024A Survey of Mamba
  4. 2025Differential Mamba
  5. 2025Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Models
  6. 2024PackMamba: Efficient Processing of Variable-Length Sequences in Mamba training
  7. 2024Falcon Mamba: The First Competitive Attention-free 7B Language Model
  8. 2025DYNAMAX: Dynamic computing for Transformers and Mamba based architectures