# FitHuBERT: Going Thinner and Deeper for Knowledge Distillation of Speech Self-Supervised Learning

arXiv: https://arxiv.org/abs/2207.00555 · HTML: https://mlsanity.com/paper/2207.00555/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation.
- Builds on 25 of its 30 references that are in the map.
- First posted on arXiv 1 Jul 2022.

## Placed in

- Knowledge distillation / [Speech and audio distillation](https://mlsanity.com/distill/speech-and-audio-distillation/)

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [fairseq: A Fast, Extensible Toolkit for Sequence Modeling](https://mlsanity.com/paper/1904.01038/) (2019)
- [SUPERB: Speech processing Universal PERformance Benchmark](https://mlsanity.com/paper/2105.01051/) (2021)
- [Listen, Attend and Spell](https://mlsanity.com/paper/1508.01211/) (2015)
- [Layer-wise Analysis of a Self-supervised Speech Representation Model](https://mlsanity.com/paper/2107.04734/) (2021)
- [TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech](https://mlsanity.com/paper/2007.06028/) (2020)
- [DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERT](https://mlsanity.com/paper/2110.01900/) (2021)
- [Lite Transformer with Long-Short Range Attention](https://mlsanity.com/paper/2004.11886/) (2020)
- [Exploring wav2vec 2.0 on speaker verification and language identification](https://mlsanity.com/paper/2012.06185/) (2020)
- [QuartzNet: Deep Automatic Speech Recognition with 1D Time-Channel Separable Convolutions](https://mlsanity.com/paper/1910.10261/) (2019)
- [Jasper: An End-to-End Convolutional Neural Acoustic Model](https://mlsanity.com/paper/1904.03288/) (2019)
- [LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERT](https://mlsanity.com/paper/2203.15610/) (2022)
- [Very Deep Self-Attention Networks for End-to-End Speech Recognition](https://mlsanity.com/paper/1904.13377/) (2019)
- [Advances in Joint CTC-Attention based End-to-End Speech Recognition with a Deep CNN Encoder and RNN-LM](https://mlsanity.com/paper/1706.02737/) (2017)
- [An Exploration of Self-Supervised Pretrained Representations for End-to-End Speech Recognition](https://mlsanity.com/paper/2110.04590/) (2021)
- [Applying Wav2vec2.0 to Speech Recognition in Various Low-resource Languages](https://mlsanity.com/paper/2012.12121/) (2020)
- [Shrinking Bigfoot: Reducing wav2vec 2.0 footprint](https://mlsanity.com/paper/2103.15760/) (2021)
- [Multi-task Voice Activated Framework using Self-supervised Learning](https://mlsanity.com/paper/2110.01077/) (2021)
- [Transformer-based ASR Incorporating Time-reduction Layer and Fine-tuning with Self-Knowledge Distillation](https://mlsanity.com/paper/2103.09903/) (2021)

## Built on by (most cited first, 25 of 25)

- [Fine-tuning Strategies for Faster Inference using Speech Self-Supervised Models: A Comparative Study](https://mlsanity.com/paper/2303.06740/) (2023)
- [Task-Agnostic Structured Pruning of Speech Representation Models](https://mlsanity.com/paper/2306.01385/) (2023)
- [CoLLD: Contrastive Layer-to-layer Distillation for Compressing Multilingual Pre-trained Speech Encoders](https://mlsanity.com/paper/2309.07707/) (2023)
- [DistillW2V2: A Small and Streaming Wav2vec 2.0 Based ASR Model](https://mlsanity.com/paper/2303.09278/) (2023)
- [Match to Win: Analysing Sequences Lengths for Efficient Self-supervised Learning in Speech and Audio](https://mlsanity.com/paper/2209.15575/) (2022)
- [Recycle-and-Distill: Universal Compression Strategy for Transformer-based Speech SSL Models with Attention Map Reusing and Masking Distillation](https://mlsanity.com/paper/2305.11685/) (2023)
- [SKILL: Similarity-aware Knowledge distILLation for Speech Self-Supervised Learning](https://mlsanity.com/paper/2402.16830/) (2024)
- [Effective and Efficient Mixed Precision Quantization of Speech Foundation Models](https://mlsanity.com/paper/2501.03643/) (2025)
- [DAISY: Data Adaptive Self-Supervised Early Exit for Speech Representation Models](https://mlsanity.com/paper/2406.05464/) (2024)
- [Sustainable self-supervised learning for speech representations](https://mlsanity.com/paper/2406.07696/) (2024)
- [One-pass Multiple Conformer and Foundation Speech Systems Compression and Quantization Using An All-in-one Neural Model](https://mlsanity.com/paper/2406.10160/) (2024)
- [On-device Streaming Discrete Speech Units](https://mlsanity.com/paper/2506.01845/) (2025)
- [STaR: Distilling Speech Temporal Relation for Lightweight Speech Self-Supervised Learning Models](https://mlsanity.com/paper/2312.09040/) (2023)
- [AdaKD: Dynamic Knowledge Distillation of ASR models using Adaptive Loss Weighting](https://mlsanity.com/paper/2405.08019/) (2024)
- [DiceHuBERT: Distilling HuBERT with a Self-Supervised Learning Objective](https://mlsanity.com/paper/2507.02911/) (2025)
- [Scaling and Distilling Transformer Models for sEMG](https://mlsanity.com/paper/2507.22094/) (2025)
- [Adaptive Knowledge Distillation for Device-Directed Speech Detection](https://mlsanity.com/paper/2508.02801/) (2025)
- [DistilXLSR: A Light Weight Cross-Lingual Speech Representation Model](https://mlsanity.com/paper/2306.01303/) (2023)
- [Efficiency-oriented approaches for self-supervised speech representation learning](https://mlsanity.com/paper/2312.11142/) (2023)
- [Effective and Efficient One-pass Compression of Speech Foundation Models Using Sparsity-aware Self-pinching Gates](https://mlsanity.com/paper/2505.22608/) (2025)
- [HARNESS: Lightweight Distilled Arabic Speech Foundation Models](https://mlsanity.com/paper/2509.14689/) (2025)
- [AdaPTwin: Low-Cost Adaptive Compression of Product Twins in Transformers](https://mlsanity.com/paper/2406.08904/) (2024)
- [Convex Distillation: Efficient Compression of Deep Networks via Convex Optimization](https://mlsanity.com/paper/2410.06567/) (2024)
- [HARNESS: Lightweight Distilled Arabic Speech Foundation Models](https://mlsanity.com/paper/2604.14186/) (2026)
- [Towards Data-free and Training-free Compression for Speech Foundation Models Using Parameter Clustering](https://mlsanity.com/paper/2606.11836/) (2026)

## Abstract (arXiv)

Large-scale speech self-supervised learning (SSL) has emerged to the main field of speech processing, however, the problem of computational cost arising from its vast size makes a high entry barrier to academia. In addition, existing distillation techniques of speech SSL models compress the model by reducing layers, which induces performance degradation in linguistic pattern recognition tasks such as phoneme recognition (PR). In this paper, we propose FitHuBERT, which makes thinner in dimension throughout almost all model components and deeper in layer compared to prior speech SSL distillation works. Moreover, we employ a time-reduction layer to speed up inference time and propose a method of hint-based distillation for less performance degradation. Our method reduces the model to 23.8% in size and 35.9% in inference time compared to HuBERT. Also, we achieve 12.1% word error rate and 13.3% phoneme error rate on the SUPERB benchmark which is superior than prior work.