mlsanityData to 7 Oct 2026

Llamba: Scaling Distilled Recurrent Models for Efficient Language Processing

arXiv 2502.14458 ↗First posted 20 Feb 2025cs.LG cs.AIcited by 35

On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language

Abstract (arXiv)

We introduce Llamba, a family of efficient recurrent language models distilled from Llama-3.x into the Mamba architecture. The series includes Llamba-1B, Llamba-3B, and Llamba-8B, which achieve higher inference throughput and handle significantly larger batch sizes than Transformer-based models while maintaining comparable benchmark performance. Furthermore, Llamba demonstrates the effectiveness of cross-architecture distillation using MOHAWK (Bick et al., 2024), achieving these results with less than 0.1% of the training data typically used for models of similar size. To take full advantage of their efficiency, we provide an optimized implementation of Llamba for resource-constrained devices such as smartphones and edge platforms, offering a practical and memory-efficient alternative to Transformers. Overall, Llamba improves the tradeoff between speed, memory efficiency, and performance, making high-quality language models more accessible.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 20 of its 27 references that are in the map. First posted on arXiv 20 Feb 2025.

Builds on

20 of 27 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Falcon Mamba: The First Competitive Attention-free 7B Language Model7 Oct
  2. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Models27 Aug
  3. 2024Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models19 Aug
  4. 2024An Empirical Study of Mamba-based Language Models12 Jun
  5. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality31 May

Other references · 15

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  5. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  6. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  7. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  8. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  9. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  10. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 700
  11. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  12. 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 208
3 more
  1. 2024Zamba: A Compact 7B SSM Hybrid Modelcited by 118
  2. 2024RNNs are not Transformers (Yet): The Key Bottleneck on In-context Retrievalcited by 62
  3. 2024LoLCATs: On Low-Rank Linearizing of Large Language Modelscited by 25

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 16; 2026: 19.

35 papers.

Sort:

By area:

  1. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  2. 2025A Survey on Latent Reasoningcited by 71
  3. 2025RADLADS: Rapid Attention Distillation to Linear Attention Decoders at Scalecited by 24
  4. 2025Thinking Slow, Fast: Scaling Inference Compute with Distilled Reasonerscited by 22
  5. 2025Zebra-Llama: Towards Extremely Efficient Hybrid Modelscited by 18
  6. 2025Citrus: Leveraging Expert Cognitive Pathways in a Medical Language Model for Advanced Medical Decision Supportcited by 16
  7. 2025Reasoning Beyond Limits: Advances and Open Problems for LLMscited by 16
  8. 2025Understanding the Skill Gap in Recurrent Language Models: The Role of the Gather-and-Aggregate Mechanismcited by 15
  9. 2025Comba: Improving Bilinear RNNs with Closed-loop Controlcited by 12
  10. 2025LoLA: Low-Rank Linear Attention With Sparse Cachingcited by 10
  11. 2025On Pruning State-Space LLMscited by 6
  12. 2026Retrieval-Aware Distillation for Transformer-SSM Hybridscited by 5
23 more
  1. 2025Data Efficient Any Transformer-to-Mamba Distillation via Attention Bridgecited by 4
  2. 2026ViT-AdaLA: Adapting Vision Transformers with Linear Attentioncited by 4
  3. 2026Raven: High-Recall Sequence Modeling with Sparse Memory Routingcited by 4
  4. 2025SpikingMamba: Towards Energy-Efficient Large Language Models via Knowledge Distillation from Mambacited by 3
  5. 2026Effective Distillation to Hybrid xLSTM Architecturescited by 3
  6. 2026Attention to Mamba: A Recipe for Cross-Architecture Distillationcited by 3
  7. 2025Technologies on Effectiveness and Efficiency: A Survey of State Spaces Modelscited by 2
  8. 2025Apriel-H1: Towards Efficient Enterprise Reasoning Modelscited by 2
  9. 2026SpikingBrain2.0: Brain-Inspired Foundation Models for Efficient Long-Context and Cross-Platform Inferencecited by 2
  10. 2026Morphing into Hybrid Attention Modelscited by 2
  11. 2026Sliding-window beats linear attentioncited by 2
  12. 2026MamTra: A Hybrid Mamba-Transformer Backbone for Speech Synthesiscited by 1
  13. 2026When Perplexity Lies: Generation-Focused Distillation of Hybrid Sequence Modelscited by 1
  14. 2026On Subquadratic Architectures: From Applications to Principlescited by 1
  15. 2026Taylor-Calibrate: Principled Initialization for Hybrid Linear Attention Distillationcited by 1
  16. 2026The Key to Going Linear: Analysis-Driven Transformer Linearizationcited by 1
  17. 2025Network of Theseus (like the ship)cited by 0
  18. 2026MAR: Efficient Large Language Models via Module-aware Architecture Refinementcited by 0
  19. 2026Towards Multiscale Graph-based Protein Learning with Geometric Secondary Structural Motifscited by 0
  20. 2026Long-Context Aware Upcycling: A New Frontier for Hybrid LLM Scalingcited by 0
  21. 2026GTR: Gated Token Recurrence for Efficient Dense Predictioncited by 0
  22. 2026On-Policy Attention Linearizationcited by 0
  23. 2026Bridging KV-Cache Quantization and Linear Attention: From Theory to Pretrained Weight Migrationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023LLM in a flash: Efficient Large Language Model Inference with Limited Memory
  2. 2025xLSTM 7B: A Recurrent LLM for Fast and Efficient Inference
  3. 2024Large Language Model Inference Acceleration: A Comprehensive Hardware Perspective
  4. 2025Pimba: A Processing-in-Memory Acceleration for Post-Transformer Large Language Model Serving
  5. 2025Speed Always Wins: A Survey on Efficient Architectures for Large Language Models
  6. 2024ReMamba: Equip Mamba with Effective Long-Sequence Modeling
  7. 2023Efficient LLM Inference on CPUs
  8. 2024LLM Inference Unveiled: Survey and Roofline Model Insights