mlsanityData to 7 Oct 2026

Joint Fine-tuning and Conversion of Pretrained Speech and Language Models towards Linear Complexity

arXiv 2410.06846 ↗First posted 9 Oct 2024cs.CL cs.AI cs.LGcited by 1

On the map: State-space sequence models (S4, Mamba) → Core SSM architectures for language

Abstract (arXiv)

Architectures such as Linformer and Mamba have recently emerged as competitive linear time replacements for transformers. However, corresponding large pretrained models are often unavailable, especially in non-text domains. To remedy this, we present a Cross-Architecture Layerwise Distillation (CALD) approach that jointly converts a transformer model to a linear time substitute and fine-tunes it to a target task. We also compare several means to guide the fine-tuning to optimally retain the desired inference capability from the original model. The methods differ in their use of the target model and the trajectory of the parameters. In a series of empirical studies on language processing, language modeling, and speech processing, we show that CALD can effectively recover the result of the original model, and that the guiding strategy contributes to the result. Some reasons for the variation are suggested.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Core SSM architectures for language, one of 10 directions in State-space sequence models (S4, Mamba). Builds on 58 of its 69 references that are in the map. First posted on arXiv 9 Oct 2024.

Builds on

58 of 69 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024The Mamba in the Llama: Distilling and Accelerating Hybrid Models27 Aug
  2. 2024Transformers to SSMs: Distilling Quadratic Knowledge to Subquadratic Models19 Aug
  3. 2024Transformers are SSMs: Generalized Models and Efficient Algorithms Through Structured State Space Duality31 May
  4. 2023Gated Linear Attention Transformers with Hardware-Efficient Training11 Dec
  5. 2023Mamba: Linear-Time Sequence Modeling with Selective State Spaces1 Dec
  6. 2021Efficiently Modeling Long Sequences with Structured State Spaces31 Oct

Other references · 52

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  8. 2020Longformer: The Long-Document Transformercited by 3,362
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  10. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  11. 2020Transformers are RNNs: Fast Autoregressive Transformers with Linear Attentioncited by 1,979
  12. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
38 more
  1. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  2. 2020Reformer: The Efficient Transformercited by 1,658
  3. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  4. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  5. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  6. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  7. 2023RWKV: Reinventing RNNs for the Transformer Eracited by 814
  8. 2020Efficient Transformers: A Surveycited by 784
  9. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  10. 2021Combining Recurrent, Convolutional, and Continuous-time Models with Linear State-Space Layerscited by 607
  11. 2023Retentive Network: A Successor to Transformer for Large Language Modelscited by 594
  12. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  13. 2020Efficient Content-Based Sparse Attention with Routing Transformerscited by 469
  14. 2023Resurrecting Recurrent Neural Networks for Long Sequencescited by 397
  15. 2021Random Feature Attentioncited by 350
  16. 2018Explicit Inductive Bias for Transfer Learning with Convolutional Networkscited by 226
  17. 2018TED-LIUM 3: twice as much data and corpus repartition for experiments on speaker adaptationcited by 224
  18. 2020SLURP: A Spoken Language Understanding Resource Packagecited by 197
  19. 2019Distilling Task-Specific Knowledge from BERT into Simple Neural Networkscited by 152
  20. 2023Fast Conformer with Linearly Scalable Attention for Efficient Speech Recognitioncited by 120
  21. 2024The Hedgehog & the Porcupine: Expressive Linear Attentions with Softmax Mimicrycited by 119
  22. 2021An Attention Free Transformercited by 111
  23. 2023LoSparse: Structured Compression of Large Language Models based on Low-Rank and Sparse Approximationcited by 103
  24. 2021Finetuning Pretrained Transformers into RNNscited by 94
  25. 2019Knowledge Distillation from Internal Representationscited by 87
  26. 2019DELTA: DEep Learning Transfer using Feature Map with Attention for Convolutional Networkscited by 80
  27. 2021Annealing Knowledge Distillationcited by 61
  28. 2024Mamba in Speech: Towards an Alternative to Self-Attentioncited by 58
  29. 2024Linearizing Large Language Modelscited by 51
  30. 2024Dual-path Mamba: Short and Long-term Bidirectional Selective Structured State Space Models for Speech Separationcited by 45
  31. 2024SSAMBA: Self-Supervised Audio Representation Learning with Mamba State Space Modelcited by 36
  32. 2022Fine-Tuning Pre-trained Transformers into Decaying Fast Weightscited by 26
  33. 2024Audio Mamba: Selective State Spaces for Self-Supervised Audio Representationscited by 19
  34. 2023Continuation Path Learning for Homotopy Optimizationcited by 12
  35. 2023Investigating End-to-End ASR Architectures for Long Form Audio Transcriptioncited by 11
  36. 2024DASS: Distilled Audio State Space Models Are Stronger and More Duration-Scalable Learnerscited by 7
  37. 2023Exploring RWKV for Memory Efficient and Low Latency Streaming ASRcited by 5
  38. 2023The Interpreter Understands Your Meaning: End-to-end Spoken Language Understanding Aided by Speech Translationcited by 4

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Attention to Mamba: A Recipe for Cross-Architecture Distillationcited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2023Sparse Fine-tuning for Inference Acceleration of Large Language Models
  2. 2023Towards Better Parameter-Efficient Fine-Tuning for Large Language Models: A Position Paper
  3. 2024Self-Distillation Bridges Distribution Gap in Language Model Fine-Tuning
  4. 2025Eliciting Fine-Tuned Transformer Capabilities via Inference-Time Techniques
  5. 2025Speed Always Wins: A Survey on Efficient Architectures for Large Language Models
  6. 2024Scalable Fine-tuning from Multiple Data Sources: A First-Order Approximation Approach
  7. 2026SAPE: Sandwich Adapters for Parameter Efficiency in Large Language Model Fine-Tuning
  8. 2023Tuning LayerNorm in Attention: Towards Efficient Multi-Modal LLM Finetuning