mlsanityData to 7 Oct 2026

Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models

arXiv 2408.10189 ↗First posted 19 Aug 2024cs.LG cs.AIcited by 66

On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language

Abstract (arXiv)

Transformer architectures have become a dominant paradigm for domains like language modeling but suffer in many inference settings due to their quadratic-time self-attention. Recently proposed subquadratic architectures, such as Mamba, have shown promise, but have been pretrained with substantially less computational resources than the strongest Transformer models. In this work, we present a method that is able to distill a pretrained Transformer architecture into alternative architectures such as state space models (SSMs). The key idea to our approach is that we can view both Transformers and SSMs as applying different forms of mixing matrices over the token sequences. We can thus progressively distill the Transformer architecture by matching different degrees of granularity in the SSM: first matching the mixing matrices themselves, then the hidden units at each block, and finally the end-to-end predictions. Our method, called MOHAWK, is able to distill a Mamba-2 variant based on the Phi-1.5 architecture (Phi-Mamba) using only 3B tokens and a hybrid version (Hybrid Phi-Mamba) using 5B tokens. Despite using less than 1% of the training data typically used to train models from scratch, Phi-Mamba boasts substantially stronger performance compared to all past open-source non-Transformer models. MOHAWK allows models like SSMs to leverage computational resources invested in training Transformer-based architectures, highlighting a new avenue for building such models.

Key facts. Cited by 66 papers in the map (data through 7 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 42 of its 55 references that are in the map. First posted on arXiv 19 Aug 2024.

Builds on

42 of 55 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Hydra: Bidirectional State Space Models Through Generalized Matrix Mixers13 Jul
  2. 2024Samba: Simple Hybrid State Space Models for Efficient Unlimited Context Language Modeling11 Jun
  3. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality31 May
  4. 2024Jamba: A Hybrid Transformer-Mamba Language Model28 Mar
  5. 2023Gated Linear Attention Transformers with Hardware-Efficient Training11 Dec
  6. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spaces1 Dec
  7. 2021Efficiently Modeling Long Sequences with Structured State Spaces31 Oct

Other references · 35

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  9. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  10. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  11. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  12. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
23 more
  1. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  2. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  3. 2022Hungry Hungry Hippos: Towards Language Modeling with State Space Modelscited by 495
  4. 2023Ring Attention with Blockwise Transformers for Near-Infinite Contextcited by 448
  5. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  6. 2024xLSTM: Extended Long Short-Term Memorycited by 320
  7. 2022Long Range Language Modeling via Gated State Spacescited by 265
  8. 2024MambaVision: A Hybrid Mamba-Transformer Vision Backbonecited by 222
  9. 2024HGRN2: Gated Linear RNNs with State Expansioncited by 122
  10. 2019Understanding Knowledge Distillation in Non-autoregressive Machine Translationcited by 121
  11. 2024The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicrycited by 119
  12. 2023Selective Structured State-Spaces for Long-Form Video Understandingcited by 105
  13. 2019Multilingual Neural Machine Translation with Knowledge Distillationcited by 95
  14. 2021Finetuning Pretrained Transformers into RNNscited by 94
  15. 2019Self-Knowledge Distillation in Natural Language Processingcited by 60
  16. 2024Linearizing Large Language Modelscited by 51
  17. 2024What does the Knowledge Neuron Thesis Have to do with Knowledge?cited by 49
  18. 2019Model Compression with Two-stage Multi-teacher Knowledge Distillation for Web Question Answering Systemcited by 43
  19. 2023Toeplitz Neural Network for Sequence Modelingcited by 43
  20. 2024PoinTramba: A Hybrid Transformer-Mamba Framework for Point Cloud Analysiscited by 26
  21. 2023HomoDistil: Homotopic Task-Agnostic Distillation of Pre-trained Transformerscited by 25
  22. 2018Attention-Guided Answer Distillation for Machine Reading Comprehensioncited by 18
  23. 2019TextKD-GAN: Text Generation using KnowledgeDistillation and Generative Adversarial Networkscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 39; 2026: 21.

66 papers.

Sort:

By area:

  1. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 188
  2. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  3. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Modelscited by 120
  4. 2024Gated Slot Attention for Efficient Linear-Time Sequence Modelingcited by 73
  5. 2025Dynamic Chunking for End-to-End Hierarchical Sequence Modelingcited by 65
  6. 2025End-to-End Test-Time Training for Long Contextcited by 53
  7. 2025Jet-Nemotron: Efficient Language Model with Post Neural Architecture Searchcited by 39
  8. 2025Llamba: Scaling Distilled Recurrent Models for Efficient Language Processingcited by 35
  9. 2025Liger: Linearizing Large Language Models to Gated Recurrent Structurescited by 32
  10. 2025RADLADS: Rapid Attention Distillation to Linear Attention Decoders at Scalecited by 24
  11. 2025Thinking Slow, Fast: Scaling Inference Compute with Distilled Reasonerscited by 22
  12. 2025M1: Towards Scalable Test-Time Compute with Mamba Reasoning Modelscited by 21
38 more
  1. 2025R-Sparse: Rank-Aware Activation Sparsity for Efficient LLM Inferencecited by 18
  2. 2025Zebra-Llama: Towards Extremely Efficient Hybrid Modelscited by 18
  3. 2026Hybrid Linear Attention Done Right: Efficient Distillation and Effective Architectures for Extremely Long Contextscited by 16
  4. 2025Understanding the Skill Gap in Recurrent Language Models: The Role of the Gather-and-Aggregate Mechanismcited by 15
  5. 2025Hybrid Architectures for Language Models: Systematic Analysis and Design Insightscited by 13
  6. 2025Comba: Improving Bilinear RNNs with Closed-loop Controlcited by 12
  7. 2025Exploring Diffusion Transformer Designs via Graftingcited by 12
  8. 2024LightTransfer: Your Long-Context LLM is Secretly a Hybrid Model with Effortless Adaptationcited by 11
  9. 2025Multimodal Mamba: Decoder-only Multimodal State Space Model via Quadratic to Linear Distillationcited by 11
  10. 2025LoLA: Low-Rank Linear Attention With Sparse Cachingcited by 10
  11. 2025Advancing Intelligent Sequence Modeling: Evolution, Trade-offs, and Applications of State-Space Architectures from S4 to Mambacited by 9
  12. 2024Autocorrelation Matters: Understanding the Role of Initialization Schemes for State Space Modelscited by 8
  13. 2025RAD: Redundancy-Aware Distillation for Hybrid Models via Self-Speculative Decodingcited by 8
  14. 2025Bolmo: Byteifying the Next Generation of Language Modelscited by 8
  15. 2025TransMamba: Fast Universal Architecture Adaption from Transformers to Mambacited by 6
  16. 2025On Pruning State-Space LLMscited by 6
  17. 2025MaTVLM: Hybrid Mamba-Transformer for Efficient Vision-Language Modelingcited by 6
  18. 2025ViT-Linearizer: Distilling Quadratic Knowledge into Linear-Time Vision Modelscited by 5
  19. 2026Retrieval-Aware Distillation for Transformer-SSM Hybridscited by 5
  20. 2025X-EcoMLA: Upcycling Pre-Trained Attention into MLA for Efficient and Extreme KV Compressioncited by 4
  21. 2025Data Efficient Any Transformer-to-Mamba Distillation via Attention Bridgecited by 4
  22. 2026Raven: High-Recall Sequence Modeling with Sparse Memory Routingcited by 4
  23. 2024OSMamba: Omnidirectional Spectral Mamba with Dual-Domain Prior Generator for Exposure Correctioncited by 3
  24. 2025LaTIM: Measuring Latent Token-to-Token Interactions in Mamba Modelscited by 3
  25. 2025On-the-Fly Adaptive Distillation of Transformer to Dual-State Linear Attentioncited by 3
  26. 2026Effective Distillation to Hybrid xLSTM Architecturescited by 3
  27. 2026Attention to Mamba: A Recipe for Cross-Architecture Distillationcited by 3
  28. 2025Technologies on Effectiveness and Efficiency: A Survey of State Spaces Modelscited by 2
  29. 2025Degrees of Freedom for Linear Attention: Distilling Softmax Attention with Optimal Feature Efficiencycited by 2
  30. 2026Linearizing Vision Transformer with Test-Time Trainingcited by 2
  31. 2026Morphing into Hybrid Attention Modelscited by 2
  32. 2026Solar Open 2 Technical Reportcited by 2
  33. 2026Sliding-window beats linear attentioncited by 2
  34. 2024Joint Fine-tuning and Conversion of Pretrained Speech and Language Models towards Linear Complexitycited by 1
  35. 2025What Matters in Linearizing Language Models? A Comparative Study of Architecture, Scale, and Task Adaptationcited by 1
  36. 2025FlashEVA: Accelerating LLM inference via Efficient Attentioncited by 1
  37. 2025Sample-Efficient Language Modeling with Linear Attention and Lightweight Enhancementscited by 1
  38. 2026MamTra: A Hybrid Mamba-Transformer Backbone for Speech Synthesiscited by 1

50 most cited of 66 All 66 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024An Empirical Study of Mamba-based Language Models
  2. 2024MoE-Mamba: Efficient Selective State Space Models with Mixture of Experts
  3. 2025TransMamba: A Sequence-Level Hybrid Transformer-Mamba Language Model
  4. 2025Pimba: A Processing-in-Memory Acceleration for Post-Transformer Large Language Model Serving
  5. 2025Mamba-Shedder: Post-Transformer Compression for Efficient Selective Structured State Space Models
  6. 2024MatMamba: A Matryoshka State Space Model
  7. 2024A Survey of Mamba
  8. 2026Global Divergence, Local Convergence: Representation Geometry in SSMs and Transformers