mlsanityData to 14 Oct 2026

Distilling HuBERT with LSTMs via Decoupled Knowledge Distillation

arXiv 2309.09920 ↗First posted 18 Sep 2023eess.AS cs.LG cs.SDcited by 4

On the map: Knowledge distillation → Speech and audio distillation

Abstract (arXiv)

Much research effort is being applied to the task of compressing the knowledge of self-supervised models, which are powerful, yet large and memory consuming. In this work, we show that the original method of knowledge distillation (and its more recently proposed extension, decoupled knowledge distillation) can be applied to the task of distilling HuBERT. In contrast to methods that focus on distilling internal features, this allows for more freedom in the network architecture of the compressed model. We thus propose to distill HuBERT's Transformer layers into an LSTM-based distilled model that reduces the number of parameters even below DistilHuBERT and at the same time shows improved performance in automatic speech recognition.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation. Builds on 17 of its 21 references that are in the map. First posted on arXiv 18 Sep 2023.

Builds on

17 of 21 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Models14 Jul
  2. 2021Knowledge Distillation for Neural Transducers from Large Self-Supervised Pre-trained Models7 Oct

Other references · 15

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  5. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  6. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 669
  7. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  8. 2022Decoupled Knowledge Distillationcited by 388
  9. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 298
  10. 2022Learning Audio-Visual Speech Representation by Masked Multimodal Cluster Predictioncited by 287
  11. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 144
  12. 2022LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERTcited by 50
3 more
  1. 2022Insights Into Deep Non-linear Filters for Improved Multi-channel Speech Enhancementcited by 14
  2. 2016Knowledge Distillation for Small-footprint Highway Networkscited by 13
  3. 2023Leveraging Semantic Information for Efficient Self-Supervised Emotion Recognition with Audio-Textual Distilled Modelscited by 6

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 3.

4 papers.

  1. 2025Effective and Efficient Mixed Precision Quantization of Speech Foundation Modelscited by 5
  2. 2024One-pass Multiple Conformer and Foundation Speech Systems Compression and Quantization Using An All-in-one Neural Modelcited by 4
  3. 2025On-device Streaming Discrete Speech Unitscited by 4
  4. 2025Effective and Efficient One-pass Compression of Speech Foundation Models Using Sparsity-aware Self-pinching Gatescited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025DiceHuBERT: Distilling HuBERT with a Self-Supervised Learning Objective
  2. 2024SKILL: Similarity-aware Knowledge distILLation for Speech Self-Supervised Learning
  3. 2021Knowledge distillation from language model to acoustic model: a hierarchical multi-task learning approach
  4. 2024Keep Decoding Parallel with Effective Knowledge Distillation from Language Models to End-to-end Speech Recognisers
  5. 2023Improving Knowledge Distillation for BERT Models: Loss Functions, Mapping Methods, and Weight Tuning
  6. 2025Synergistic Effects of Knowledge Distillation and Structured Pruning for Self-Supervised Speech Models
  7. 2023DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Models
  8. 2021ERNIE-Tiny : A Progressive Distillation Framework for Pretrained Transformer Compression