mlsanityData to 14 Oct 2026

DiceHuBERT: Distilling HuBERT with a Self-Supervised Learning Objective

arXiv 2507.02911 ↗First posted 25 Jun 2025cs.LG cs.AI cs.SDcited by 3

On the map: Knowledge distillation → Speech and audio distillation

Abstract (arXiv)

We introduce DiceHuBERT, a knowledge distillation framework for compressing HuBERT, a widely used self-supervised learning (SSL)-based speech foundation model. Unlike existing distillation methods that rely on layer-wise and feature-wise mapping between teacher and student models, DiceHuBERT leverages HuBERT's iterative self-distillation mechanism by directly replacing the original model with a student model. This replacement allows the student to be trained using the same SSL objective used when pre-training HuBERT, eliminating the need for additional modules or architectural constraints. Experimental results on SUPERB show that DiceHuBERT consistently outperforms existing distillation methods, improving phoneme recognition performance by over 21% and ASR performance by more than 14%. Furthermore, DiceHuBERT demonstrates competitive performance across multiple tasks, highlighting its clear advantage.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation. Builds on 24 of its 31 references that are in the map. First posted on arXiv 25 Jun 2025.

Builds on

24 of 31 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Distil-Whisper: Robust Knowledge Distillation via Large-Scale Pseudo Labelling1 Nov
  2. 2022FitHuBERT: Going Thinner and Deeper for Knowledge Distillation of Speech Self-Supervised Learning1 Jul

Other references · 22

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  5. 2020Knowledge Distillation: A Surveycited by 1,603
  6. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  7. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
  8. 2013Do Deep Nets Really Need to be Deep?cited by 474
  9. 2019Libri-Light: A Benchmark for ASR with Limited or No Supervisioncited by 467
  10. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  11. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 144
  12. 2022Knowledge Distillation with the Reused Teacher Classifiercited by 141
10 more
  1. 2020Distilling Knowledge from Graph Convolutional Networkscited by 129
  2. 2022LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERTcited by 50
  3. 2021An Exploration of Self-Supervised Pretrained Representations for End-to-End Speech Recognitioncited by 39
  4. 2023DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Modelscited by 36
  5. 2022Investigating self-supervised learning for speech enhancement and separationcited by 35
  6. 2020Why distillation helps: a statistical perspectivecited by 21
  7. 2019Speech-XLNet: Unsupervised Acoustic Model Pretraining For Self-Attention Networkscited by 17
  8. 2023Recycle-and-Distill: Universal Compression Strategy for Transformer-based Speech SSL Models with Attention Map Reusing and Masking Distillationcited by 6
  9. 2023STaR: Distilling Speech Temporal Relation for Lightweight Speech Self-Supervised Learning Modelscited by 3
  10. 2024Exploring Prediction Targets in Masked Pre-Training for Speech Foundation Modelscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026S-SONDO: Self-Supervised Knowledge Distillation for General Audio Foundation Modelscited by 0
  2. 2026Fast Speech Foundation Model Distillation Using Interleaved Stackingcited by 0
  3. 2026CAFE: Confining Feature Distillation to Language-Routed Adapters for Compact Multilingual Automatic Speech Recognitioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Models
  2. 2024SKILL: Similarity-aware Knowledge distILLation for Speech Self-Supervised Learning
  3. 2023Distilling HuBERT with LSTMs via Decoupled Knowledge Distillation
  4. 2021Knowledge Distillation for Neural Transducers from Large Self-Supervised Pre-trained Models
  5. 2023Ensemble knowledge distillation of self-supervised speech models
  6. 2023Stable Distillation: Regularizing Continued Pre-training for Low-Resource Automatic Speech Recognition
  7. 2023One-Step Knowledge Distillation and Fine-Tuning in Using Large Pre-Trained Self-Supervised Learning Models for Speaker Verification
  8. 2026ASKD-Whisper: Adaptive Self-knowledge Distillation for Efficient and Low-Latency Automatic Speech Recognition