mlsanityData to 7 Oct 2026

Ensemble knowledge distillation of self-supervised speech models

arXiv 2302.12757 ↗First posted 24 Feb 2023eess.AS cs.CL cs.SDcited by 10

Not on the map. Why?

Abstract (arXiv)

Distilled self-supervised models have shown competitive performance and efficiency in recent years. However, there is a lack of experience in jointly distilling multiple self-supervised speech models. In our work, we performed Ensemble Knowledge Distillation (EKD) on various self-supervised speech models such as HuBERT, RobustHuBERT, and WavLM. We tried two different aggregation techniques, layerwise-average and layerwise-concatenation, to the representations of different teacher models and found that the former was more effective. On top of that, we proposed a multiple prediction head method for student models to predict different layer outputs of multiple teacher models simultaneously. The experimental results show that our method improves the performance of the distilled models on four downstream speech processing tasks, Phoneme Recognition, Speaker Identification, Emotion Recognition, and Automatic Speech Recognition in the hidden-set track of the SUPERB benchmark.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 18 of its 24 references that are in the map. First posted on arXiv 24 Feb 2023.

Builds on

18 of 24 references.

References · 18

  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  3. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  4. 2015MUSAN: A Music, Speech, and Noise Corpuscited by 400
  5. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  6. 2019WHAM!: Extending Speech Separation to Noisy Environmentscited by 154
  7. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 129
  8. 2022ICASSP 2022 Deep Noise Suppression Challengecited by 57
  9. 2021One Teacher is Enough? Pre-trained Language Model Distillation from Multiple Teacherscited by 39
  10. 2022Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Modelscited by 28
  11. 2022SUPERB @ SLT 2022: Challenge on Generalization and Efficiency of Self-Supervised Speech Representation Learningcited by 26
  12. 2022Improving Distortion Robustness of Self-supervised Speech Processing Tasks with Domain Adaptationcited by 16
6 more
  1. 2021Wav2vec-Switch: Contrastive Learning from Original-noisy Speech Pairs for Robust Speech Recognitioncited by 15
  2. 2021Improving Noise Robustness of Contrastive Speech Representation Learning with Speech Reconstructioncited by 12
  3. 2020Distilling Knowledge from Ensembles of Acoustic Models for Joint CTC-Attention End-to-End Speech Recognitioncited by 6
  4. 2022Investigation of Ensemble features of Self-Supervised Pretrained Models for Automatic Speech Recognitioncited by 6
  5. 2022Unified and Effective Ensemble Knowledge Distillationcited by 5
  6. 2022Improving generalizability of distilled self-supervised speech processing models under distorted settingscited by 5

Built on by

Cited by year

Citing papers per year: 2023: 4; 2024: 3; 2025: 3.

10 papers.

  1. 2023Zero Resource Code-switched Speech Benchmark Using Speech Utterance Pairs For Multiple Spoken Languagescited by 12
  2. 2023CoLLD: Contrastive Layer-to-layer Distillation for Compressing Multilingual Pre-trained Speech Encoderscited by 7
  3. 2025USAD: Universal Speech and Audio Representation via Distillationcited by 7
  4. 2024Pheme: Efficient and Conversational Speech Generationcited by 6
  5. 2023On-Device Constrained Self-Supervised Speech Representation Learning for Keyword Spotting via Knowledge Distillationcited by 2
  6. 2024MIDAS: Multi-level Intent, Domain, And Slot Knowledge Distillation for Multi-turn NLUcited by 2
  7. 2024How Redundant Is the Transformer Stack in Speech Representation Models?cited by 2
  8. 2025Speech-FT: Merging Pre-trained And Fine-Tuned Speech Representation Models For Cross-Task Generalizationcited by 2
  9. 2025Multi-Teacher Language-Aware Knowledge Distillation for Multilingual Speech Emotion Recognitioncited by 2
  10. 2023MUST: A Multilingual Student-Teacher Learning approach for low-resource speech recognitioncited by 0