mlsanityData to 14 Oct 2026

Application of Knowledge Distillation to Multi-task Speech Representation Learning

arXiv 2210.16611 ↗First posted 29 Oct 2022eess.AS cs.CL cs.SDcited by 0

On the map: Knowledge distillation → Speech and audio distillation

Abstract (arXiv)

Model architectures such as wav2vec 2.0 and HuBERT have been proposed to learn speech representations from audio waveforms in a self-supervised manner. When they are combined with downstream tasks such as keyword spotting and speaker verification, they provide state-of-the-art performance. However, these models use a large number of parameters, the smallest version of which has 95 million parameters. This constitutes a challenge for edge AI device deployments. In this paper, we investigate the application of knowledge distillation to speech representation learning (SRL) models followed by joint fine-tuning with multiple downstream voice-activated tasks. In our experiments on two such tasks, our approach results in nearly 75% reduction in model size while suffering only 0.1% accuracy and 0.9% equal error rate degradation compared to the full-size model. In addition, we show that fine-tuning the SRL models results in a significant performance boost compared to using frozen SRL models.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation. Builds on 22 of its 29 references that are in the map. First posted on arXiv 29 Oct 2022.

Builds on

22 of 29 references.

References · 22

  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  3. 2020Knowledge Distillation: A Surveycited by 1,603
  4. 2018Speech Commands: A Dataset for Limited-Vocabulary Speech Recognitioncited by 752
  5. 2017VoxCeleb: a large-scale speaker identification datasetcited by 732
  6. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  7. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 340
  8. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 298
  9. 2019Mockingjay: Unsupervised Speech Representation Learning with Deep Bidirectional Transformer Encoderscited by 198
  10. 2020TERA: Self-Supervised Learning of Transformer Encoder Representation for Speechcited by 188
  11. 2019An Unsupervised Autoregressive Model for Speech Representation Learningcited by 186
  12. 2021Emotion Recognition from Speech Using Wav2vec 2.0 Embeddingscited by 162
10 more
  1. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 144
  2. 2021A Fine-tuned Wav2vec 2.0/HuBERT Benchmark For Speech Emotion Recognition, Speaker Verification and Spoken Language Understandingcited by 82
  3. 2021Large-scale Self-Supervised Speech Representation Learning for Automatic Speaker Verificationcited by 78
  4. 2022Speech Emotion Recognition using Self-Supervised Featurescited by 53
  5. 2022LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERTcited by 50
  6. 2021Fine-tuning wav2vec2 for speaker recognitioncited by 39
  7. 2021An Exploration of Self-Supervised Pretrained Representations for End-to-End Speech Recognitioncited by 39
  8. 2020Deep Representation Learning in Speech Processing: Challenges, Recent Advances, and Future Trendscited by 28
  9. 2021Shrinking Bigfoot: Reducing wav2vec 2.0 footprintcited by 21
  10. 2021Multi-task Voice Activated Framework using Self-supervised Learningcited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2021Knowledge distillation from language model to acoustic model: a hierarchical multi-task learning approach
  2. 2023Knowledge Distillation from Multiple Foundation Models for End-to-End Speech Recognition
  3. 2023Knowledge Transfer from Pre-trained Language Models to Cif-based Speech Recognizers via Hierarchical Distillation
  4. 2022Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Models
  5. 2024SKILL: Similarity-aware Knowledge distILLation for Speech Self-Supervised Learning
  6. 2024Integrated Multi-Level Knowledge Distillation for Enhanced Speaker Verification
  7. 2021Knowledge Distillation for Neural Transducers from Large Self-Supervised Pre-trained Models
  8. 2023One-Step Knowledge Distillation and Fine-Tuning in Using Large Pre-Trained Self-Supervised Learning Models for Speaker Verification