mlsanityData to 7 Oct 2026

SKILL: Similarity-aware Knowledge distILLation for Speech Self-Supervised Learning

arXiv 2402.16830 ↗First posted 26 Feb 2024eess.AS cs.CL cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning (SSL) has achieved remarkable success across various speech-processing tasks. To enhance its efficiency, previous works often leverage the use of compression techniques. A notable recent attempt is DPHuBERT, which applies joint knowledge distillation (KD) and structured pruning to learn a significantly smaller SSL model. In this paper, we contribute to this research domain by introducing SKILL, a novel method that conducts distillation across groups of layers instead of distilling individual arbitrarily selected layers within the teacher network. The identification of the layers to distill is achieved through a hierarchical clustering procedure applied to layer similarity measures. Extensive experiments demonstrate that our distilled version of WavLM Base+ not only outperforms DPHuBERT but also achieves state-of-the-art results in the 30M parameters model class across several SUPERB tasks.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 35 of its 39 references that are in the map. First posted on arXiv 26 Feb 2024.

Builds on

35 of 39 references.

References · 35

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  3. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  4. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  5. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  6. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  7. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  8. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  9. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 589
  10. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
  11. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
  12. 2021SpeechBrain: A General-Purpose Speech Toolkitcited by 404
23 more
  1. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  2. 2019vq-wav2vec: Self-Supervised Learning of Discrete Speech Representationscited by 335
  3. 2021Distilling Knowledge via Knowledge Reviewcited by 285
  4. 2019Structured Pruning of Large Language Modelscited by 201
  5. 2022Structured Pruning Learns Compact and Accurate Modelscited by 167
  6. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 129
  7. 2020Multi-task self-supervised learning for Robust Speech Recognitioncited by 127
  8. 2020Exploring wav2vec 2.0 on speaker verification and language identificationcited by 93
  9. 2021LeBenchmark: A Reproducible Framework for Assessing Self-Supervised Representation Learning from Speechcited by 46
  10. 2023DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Modelscited by 32
  11. 2023Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understandingcited by 29
  12. 2022Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Modelscited by 28
  13. 2023Speech Self-Supervised Representation Benchmarking: Are We Doing it Right?cited by 28
  14. 2018Learning Speaker Representations with Mutual Informationcited by 26
  15. 2021Wav2vec-C: A Self-supervised Model for Speech Representation Learningcited by 25
  16. 2023TorchAudio 2.1: Advancing speech recognition, self-supervised learning, and audio processing components for PyTorchcited by 25
  17. 2022FitHuBERT: Going Thinner and Deeper for Knowledge Distillation of Speech Self-Supervised Learningcited by 24
  18. 2021MetricGAN-U: Unsupervised speech enhancement/ dereverberation based only on noisy/ reverberated speechcited by 18
  19. 2023Fine-tuning Strategies for Faster Inference using Speech Self-Supervised Models: A Comparative Studycited by 15
  20. 2023Speech Emotion Diarization: Which Emotion Appears When?cited by 13
  21. 2021Multi-task Voice Activated Framework using Self-supervised Learningcited by 8
  22. 2023Recycle-and-Distill: Universal Compression Strategy for Transformer-based Speech SSL Models with Attention Map Reusing and Masking Distillationcited by 6
  23. 2021DNN Quantization with Attentioncited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 1.

4 papers.

  1. 2024How Redundant Is the Transformer Stack in Speech Representation Models?cited by 2
  2. 2025Effective and Efficient One-pass Compression of Speech Foundation Models Using Sparsity-aware Self-pinching Gatescited by 1
  3. 2025HARNESS: Lightweight Distilled Arabic Speech Foundation Modelscited by 0
  4. 2026HARNESS: Lightweight Distilled Arabic Speech Foundation Modelscited by 0