mlsanityData to 7 Oct 2026

DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERT

arXiv 2110.01900 ↗First posted 5 Oct 2021cs.CL eess.AScited by 129

Not on the map. Why?

Abstract (arXiv)

Self-supervised speech representation learning methods like wav2vec 2.0 and Hidden-unit BERT (HuBERT) leverage unlabeled speech data for pre-training and offer good representations for numerous speech processing tasks. Despite the success of these methods, they require large memory and high pre-training costs, making them inaccessible for researchers in academia and small companies. Therefore, this paper introduces DistilHuBERT, a novel multi-task learning framework to distill hidden representations from a HuBERT model directly. This method reduces HuBERT's size by 75% and 73% faster while retaining most performance in ten different tasks. Moreover, DistilHuBERT required little training time and data, opening the possibilities of pre-training personal and on-device SSL models for speech.

Key facts. Cited by 129 papers in the map (data through 7 Oct 2026). Builds on 32 of its 36 references that are in the map. First posted on arXiv 5 Oct 2021.

Builds on

32 of 36 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  6. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  7. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  8. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  9. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  10. 2017AISHELL-1: An Open-Source Mandarin Speech Corpus and A Speech Recognition Baselinecited by 364
  11. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  12. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 293
20 more
  1. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 196
  2. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 186
  3. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 185
  4. 2021Unsupervised Speech Recognitioncited by 147
  5. 2020Multi-task self-supervised learning for Robust Speech Recognitioncited by 127
  6. 2020Unsupervised pretraining transfers well across languagescited by 100
  7. 2019Learning Problem-agnostic Speech Representations from Multiple Self-supervised Taskscited by 87
  8. 2020DeCoAR 2.0: Deep Contextualized Acoustic Representations with Vector Quantizationcited by 77
  9. 2019Effectiveness of self-supervised pre-training for speech recognitioncited by 72
  10. 2020Vector-Quantized Autoregressive Predictive Codingcited by 69
  11. 2019Deep Contextualized Acoustic Representations For Semi-Supervised Speech Recognitioncited by 66
  12. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 50
  13. 2020Non-Autoregressive Predictive Coding for Learning Speech Representations from Local Dependenciescited by 47
  14. 2021LeBenchmark: A Reproducible Framework for Assessing Self-Supervised Representation Learning from Speechcited by 46
  15. 2020Speech SIMCLR: Combining Contrastive and Reconstruction Objective for Self-supervised Speech Representation Learningcited by 33
  16. 2021Wav2vec-C: A Self-supervised Model for Speech Representation Learningcited by 25
  17. 2020Improved Speech Representations with Multi-Target Autoregressive Predictive Codingcited by 23
  18. 2021Towards Lifelong Learning of End-to-end ASRcited by 20
  19. 2021Shrinking Bigfoot: Reducing wav2vec 2.0 footprintcited by 16
  20. 2021The History of Speech Recognition to the Year 2030cited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 31; 2023: 36; 2024: 28; 2025: 18; 2026: 15.

129 papers.

Sort:

By area:

  1. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 718
  2. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  3. 2023Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labellingcited by 87
  4. 2023SpeechTokenizer: Unified Speech Tokenizer for Speech Large Language Modelscited by 85
  5. 2022SUPERB-SG: Enhanced Speech processing Universal PERformance Benchmark for Semantic and Generative Capabilitiescited by 76
  6. 2024OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformercited by 74
  7. 2023What Do Self-Supervised Speech Models Know About Words?cited by 62
  8. 2022BridgeTower: Building Bridges Between Encoders in Vision-Language Representation Learningcited by 56
  9. 2022LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERTcited by 45
  10. 2022On the Utility of Self-supervised Models for Prosody-related Taskscited by 42
  11. 2023Reproducing Whisper-Style Training Using an Open-Source Toolkit and Publicly Available Datacited by 42
  12. 2024A Large-Scale Evaluation of Speech Foundation Modelscited by 38
38 more
  1. 2023DinoSR: Self-Distillation and Online Clustering for Self-supervised Speech Representation Learningcited by 34
  2. 2024OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identificationcited by 34
  3. 2023DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Modelscited by 32
  4. 2022SpeechCLIP: Integrating Speech with Pre-Trained Vision and Language Modelcited by 29
  5. 2023Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understandingcited by 29
  6. 2022Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Modelscited by 28
  7. 2023Speech Self-Supervised Representation Benchmarking: Are We Doing it Right?cited by 28
  8. 2022SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learningcited by 26
  9. 2022Self-Supervised Representation Learning for Speech Using Visual Grounding and Masked Language Modelingcited by 25
  10. 2022FitHuBERT: Going Thinner and Deeper for Knowledge Distillation of Speech Self-Supervised Learningcited by 24
  11. 2023Self-supervised Fine-tuning for Improved Content Representations by Speaker-invariant Clusteringcited by 21
  12. 2023Reducing Barriers to Self-Supervised Learning: HuBERT Pre-training with Academic Computecited by 21
  13. 2025AudioLens: A Closer Look at Auditory Attribute Perception of Large Audio-Language Modelscited by 18
  14. 2023Speech Self-Supervised Representations Benchmarking: a Case for Larger Probing Headscited by 16
  15. 2024On the social bias of speech self-supervised modelscited by 16
  16. 2023Fine-tuning Strategies for Faster Inference using Speech Self-Supervised Models: A Comparative Studycited by 15
  17. 2023Accelerating Transducers through Adjacent Token Mergingcited by 14
  18. 2022HuBERT-EE: Early Exiting HuBERT for Efficient Speech Recognitioncited by 13
  19. 2023Task-Agnostic Structured Pruning of Speech Representation Modelscited by 13
  20. 2024SSDM: Scalable Speech Dysfluency Modelingcited by 13
  21. 2022Semi-FedSER: Semi-supervised Learning for Speech Emotion Recognition On Federated Learning using Multiview Pseudo-Labelingcited by 12
  22. 2023Let There Be Sound: Reconstructing High Quality Speech from Silent Videoscited by 12
  23. 2023Ensemble knowledge distillation of self-supervised speech modelscited by 10
  24. 2023I3D: Transformer architectures with input-dependent dynamic depth for speech recognitioncited by 10
  25. 2023DQ-Whisper: Joint Distillation and Quantization for Efficient Multilingual Speech Recognitioncited by 10
  26. 2023Accurate and Structured Pruning for Efficient Automatic Speech Recognitioncited by 10
  27. 2023Can Self-Supervised Neural Representations Pre-Trained on Human Speech distinguish Animal Callers?cited by 9
  28. 2024DP-DyLoRA: Fine-Tuning Transformer-Based Models On-Device under Differentially Private Federated Learning using Dynamic Low-Rank Adaptationcited by 9
  29. 2022Learning Lip-Based Audio-Visual Speaker Embeddings with AV-HuBERTcited by 8
  30. 2024DC-Spin: A Speaker-invariant Speech Tokenizer for Spoken Language Modelscited by 8
  31. 2022Silence is Sweeter Than Speech: Self-Supervised Model Using Silence to Store Speaker Informationcited by 7
  32. 2022On Compressing Sequences for Self-Supervised Speech Modelscited by 7
  33. 2023CoLLD: Contrastive Layer-to-layer Distillation for Compressing Multilingual Pre-trained Speech Encoderscited by 7
  34. 2023R-Spin: Efficient Speaker and Noise-invariant Representation Learning with Acoustic Piecescited by 7
  35. 2025USAD: Universal Speech and Audio Representation via Distillationcited by 7
  36. 2022Enhancing and Adversarial: Improve ASR with Speaker Labelscited by 6
  37. 2023Recycle-and-Distill: Universal Compression Strategy for Transformer-based Speech SSL Models with Attention Map Reusing and Masking Distillationcited by 6
  38. 2023Leveraging Semantic Information for Efficient Self-Supervised Emotion Recognition with Audio-Textual Distilled Modelscited by 6

50 most cited of 129 All 129 (JSON)