mlsanityData to 14 Oct 2026

FitHuBERT: Going Thinner and Deeper for Knowledge Distillation of Speech Self-Supervised Learning

arXiv 2207.00555 ↗First posted 1 Jul 2022eess.AS cs.CL cs.LGcited by 25

On the map: Knowledge distillation → Speech and audio distillation

Abstract (arXiv)

Large-scale speech self-supervised learning (SSL) has emerged to the main field of speech processing, however, the problem of computational cost arising from its vast size makes a high entry barrier to academia. In addition, existing distillation techniques of speech SSL models compress the model by reducing layers, which induces performance degradation in linguistic pattern recognition tasks such as phoneme recognition (PR). In this paper, we propose FitHuBERT, which makes thinner in dimension throughout almost all model components and deeper in layer compared to prior speech SSL distillation works. Moreover, we employ a time-reduction layer to speed up inference time and propose a method of hint-based distillation for less performance degradation. Our method reduces the model to 23.8% in size and 35.9% in inference time compared to HuBERT. Also, we achieve 12.1% word error rate and 13.3% phoneme error rate on the SUPERB benchmark which is superior than prior work.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation. Builds on 25 of its 30 references that are in the map. First posted on arXiv 1 Jul 2022.

Builds on

25 of 30 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Transformer-based ASR Incorporating Time-reduction Layer and Fine-tuning with Self-Knowledge Distillation17 Mar

Other references · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  6. 2013Visualizing and Understanding Convolutional Networkscited by 2,290
  7. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  8. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  9. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  10. 2015Listen, Attend and Spellcited by 543
  11. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 298
  12. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 188
12 more
  1. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 144
  2. 2020Lite Transformer with Long-Short Range Attentioncited by 127
  3. 2020Exploring wav2vec 2.0 on speaker verification and language identificationcited by 95
  4. 2019QuartzNet: Deep Automatic Speech Recognition with 1D Time-Channel Separable Convolutionscited by 94
  5. 2019Jasper: An End-to-End Convolutional Neural Acoustic Modelcited by 76
  6. 2022LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERTcited by 50
  7. 2019Very Deep Self-Attention Networks for End-to-End Speech Recognitioncited by 46
  8. 2017Advances in Joint CTC-Attention based End-to-End Speech Recognition with a Deep CNN Encoder and RNN-LMcited by 43
  9. 2021An Exploration of Self-Supervised Pretrained Representations for End-to-End Speech Recognitioncited by 39
  10. 2020Applying Wav2vec2.0 to Speech Recognition in Various Low-resource Languagescited by 30
  11. 2021Shrinking Bigfoot: Reducing wav2vec 2.0 footprintcited by 21
  12. 2021Multi-task Voice Activated Framework using Self-supervised Learningcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 8; 2024: 7; 2025: 7; 2026: 2.

25 papers.

Sort:

By area:

  1. 2023Fine-tuning Strategies for Faster Inference using Speech Self-Supervised Models: A Comparative Studycited by 15
  2. 2023Task-Agnostic Structured Pruning of Speech Representation Modelscited by 15
  3. 2023CoLLD: Contrastive Layer-to-layer Distillation for Compressing Multilingual Pre-trained Speech Encoderscited by 9
  4. 2023DistillW2V2: A Small and Streaming Wav2vec 2.0 Based ASR Modelcited by 7
  5. 2022Match to Win: Analysing Sequences Lengths for Efficient Self-supervised Learning in Speech and Audiocited by 6
  6. 2023Recycle-and-Distill: Universal Compression Strategy for Transformer-based Speech SSL Models with Attention Map Reusing and Masking Distillationcited by 6
  7. 2024SKILL: Similarity-aware Knowledge distILLation for Speech Self-Supervised Learningcited by 5
  8. 2025Effective and Efficient Mixed Precision Quantization of Speech Foundation Modelscited by 5
  9. 2024DAISY: Data Adaptive Self-Supervised Early Exit for Speech Representation Modelscited by 4
  10. 2024Sustainable self-supervised learning for speech representationscited by 4
  11. 2024One-pass Multiple Conformer and Foundation Speech Systems Compression and Quantization Using An All-in-one Neural Modelcited by 4
  12. 2025On-device Streaming Discrete Speech Unitscited by 4
13 more
  1. 2023STaR: Distilling Speech Temporal Relation for Lightweight Speech Self-Supervised Learning Modelscited by 3
  2. 2024AdaKD: Dynamic Knowledge Distillation of ASR models using Adaptive Loss Weightingcited by 3
  3. 2025DiceHuBERT: Distilling HuBERT with a Self-Supervised Learning Objectivecited by 3
  4. 2025Scaling and Distilling Transformer Models for sEMGcited by 3
  5. 2025Adaptive Knowledge Distillation for Device-Directed Speech Detectioncited by 3
  6. 2023DistilXLSR: A Light Weight Cross-Lingual Speech Representation Modelcited by 2
  7. 2023Efficiency-oriented approaches for self-supervised speech representation learningcited by 1
  8. 2025Effective and Efficient One-pass Compression of Speech Foundation Models Using Sparsity-aware Self-pinching Gatescited by 1
  9. 2025HARNESS: Lightweight Distilled Arabic Speech Foundation Modelscited by 1
  10. 2024AdaPTwin: Low-Cost Adaptive Compression of Product Twins in Transformerscited by 0
  11. 2024Convex Distillation: Efficient Compression of Deep Networks via Convex Optimizationcited by 0
  12. 2026HARNESS: Lightweight Distilled Arabic Speech Foundation Modelscited by 0
  13. 2026Towards Data-free and Training-free Compression for Speech Foundation Models Using Parameter Clusteringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Models
  2. 2022Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Models
  3. 2023Stable Distillation: Regularizing Continued Pre-training for Low-Resource Automatic Speech Recognition
  4. 2021Knowledge Distillation for Neural Transducers from Large Self-Supervised Pre-trained Models
  5. 2022Comparison of Soft and Hard Target RNN-T Distillation for Large-scale ASR
  6. 2023Distilling HuBERT with LSTMs via Decoupled Knowledge Distillation
  7. 2022Improving generalizability of distilled self-supervised speech processing models under distorted settings
  8. 2023Ensemble knowledge distillation of self-supervised speech models