mlsanityData to 14 Oct 2026

Synergistic Effects of Knowledge Distillation and Structured Pruning for Self-Supervised Speech Models

arXiv 2502.05837 ↗First posted 9 Feb 2025eess.AScited by 0

On the map: Knowledge distillation → Speech and audio distillation

Abstract (arXiv)

Traditionally, Knowledge Distillation (KD) is used for model compression, often leading to suboptimal performance. In this paper, we evaluate the impact of combining KD loss with alternative pruning techniques, including Low-Rank Factorization (LRF) and l0 regularization, on a conformer-based pre-trained network under the paradigm of Self-Supervised Learning (SSL). We also propose a strategy to jointly prune and train an RNN-T-based ASR model, demonstrating that this approach yields superior performance compared to pruning a pre-trained network first and then using it for ASR training. This approach led to a significant reduction in word error rate: l0 and KD combination achieves the best non-streaming performance, with a 8.9% Relative Word Error Rate (RWER) improvement over the baseline, while LRF and KD combination yields the best results for streaming ASR, improving RWER by 13.4%.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Speech and audio distillation, one of 12 directions in Knowledge distillation. Builds on 24 of its 31 references that are in the map. First posted on arXiv 9 Feb 2025.

Builds on

24 of 31 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023On the compression of shallow non-causal ASR models using knowledge distillation and tied-and-reduced decoder for low-latency on-device speech recognition15 Dec
  2. 2022Multi-stage Progressive Compression of Conformer Transducer for On-device Speech Recognition1 Oct
  3. 2022Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Models14 Jul

Other references · 21

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  3. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  4. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,918
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  6. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,863
  7. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  8. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,380
  9. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 787
  10. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  11. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 583
  12. 2021VoxPopuli: A Large-Scale Multilingual Speech Corpus for Representation Learning, Semi-Supervised Learning and Interpretationcited by 463
9 more
  1. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 347
  2. 2022Self-supervised Learning with Random-projection Quantizer for Speech Recognitioncited by 162
  3. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 144
  4. 2020Cascaded encoders for unifying streaming and non-streaming ASRcited by 53
  5. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 53
  6. 2022LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERTcited by 50
  7. 2020Efficient Knowledge Distillation for RNN-Transducer Modelscited by 38
  8. 2023DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Modelscited by 36
  9. 2018DeepTwist: Learning Model Compression via Occasional Weight Distortioncited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2021Knowledge Distillation for Neural Transducers from Large Self-Supervised Pre-trained Models
  2. 2023Accurate and Structured Pruning for Efficient Automatic Speech Recognition
  3. 2024SKILL: Similarity-aware Knowledge distILLation for Speech Self-Supervised Learning
  4. 2025Fine-tune Before Structured Pruning: Towards Compact and Accurate Self-Supervised Models for Speaker Diarization
  5. 2023Knowledge Distillation from Multiple Foundation Models for End-to-End Speech Recognition
  6. 2025Hybrid Pruning: In-Situ Compression of Self-Supervised Speech Models for Speaker Verification and Anti-Spoofing
  7. 2024Leave No Knowledge Behind During Knowledge Distillation: Towards Practical and Effective Knowledge Distillation for Code-Switching ASR Using Realistic Data
  8. 2022Comparison of Soft and Hard Target RNN-T Distillation for Large-scale ASR