mlsanityData to 14 Oct 2026

DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Models

arXiv 2305.17651 ↗First posted 28 May 2023cs.CL cs.SD eess.AScited by 36

Not on the map. Why?

Abstract (arXiv)

Self-supervised learning (SSL) has achieved notable success in many speech processing tasks, but the large model size and heavy computational cost hinder the deployment. Knowledge distillation trains a small student model to mimic the behavior of a large teacher model. However, the student architecture usually needs to be manually designed and will remain fixed during training, which requires prior knowledge and can lead to suboptimal performance. Inspired by recent success of task-specific structured pruning, we propose DPHuBERT, a novel task-agnostic compression method for speech SSL based on joint distillation and pruning. Experiments on SUPERB show that DPHuBERT outperforms pure distillation methods in almost all tasks. Moreover, DPHuBERT requires little training time and performs well with limited training data, making it suitable for resource-constrained applications. Our method can also be applied to various speech SSL models. Our code and models will be publicly available.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Builds on 35 of its 42 references that are in the map. First posted on arXiv 28 May 2023.

Builds on

35 of 42 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  5. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  6. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  7. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  8. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  9. 2019Are Sixteen Heads Really Better than One?cited by 816
  10. 2019Once-for-All: Train One Network and Specialize it for Efficient Deploymentcited by 787
  11. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  12. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
23 more
  1. 2021XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scalecited by 601
  2. 2018Rethinking the Value of Network Pruningcited by 599
  3. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 583
  4. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
  5. 2021Layer-wise Analysis of a Self-supervised Speech Representation Modelcited by 298
  6. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 260
  7. 2019Structured Pruning of Large Language Modelscited by 203
  8. 2022Structured Pruning Learns Compact and Accurate Modelscited by 170
  9. 2021Unsupervised Speech Recognitioncited by 148
  10. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 144
  11. 2022Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understandingcited by 91
  12. 2021TorchAudio: Building Blocks for Audio and Speech Processingcited by 83
  13. 2021Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognitioncited by 68
  14. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 53
  15. 2022LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERTcited by 50
  16. 2021An Exploration of Self-Supervised Pretrained Representations for End-to-End Speech Recognitioncited by 39
  17. 2022Investigating self-supervised learning for speech enhancement and separationcited by 35
  18. 2022Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Modelscited by 31
  19. 2023Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understandingcited by 31
  20. 2020On the Usefulness of Self-Attention for Automatic Speech Recognition with Transformerscited by 24
  21. 2020RTMobile: Beyond Real-Time Mobile Acceleration of RNNs for Speech Recognitioncited by 19
  22. 2022A Study on the Integration of Pre-trained SSL, ASR, LM and SLU Models for Spoken Language Understandingcited by 18
  23. 2022MelHuBERT: A simplified HuBERT on Mel spectrogramscited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 14; 2025: 12; 2026: 5.

36 papers.

Sort:

By area:

  1. 2025On The Landscape of Spoken Language Models: A Comprehensive Surveycited by 118
  2. 2024OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformercited by 74
  3. 2023Reproducing Whisper-Style Training Using an Open-Source Toolkit and Publicly Available Datacited by 42
  4. 2024A Large-Scale Evaluation of Speech Foundation Modelscited by 40
  5. 2024OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identificationcited by 37
  6. 2024SSDM: Scalable Speech Dysfluency Modelingcited by 13
  7. 2023CoLLD: Contrastive Layer-to-layer Distillation for Compressing Multilingual Pre-trained Speech Encoderscited by 9
  8. 2025Efficient and Robust Speaker Diarization via Structured Pruning of Self-Supervised Modelscited by 7
  9. 2025USAD: Universal Speech and Audio Representation via Distillationcited by 7
  10. 2024Pheme: Efficient and Conversational Speech Generationcited by 6
  11. 2024SKILL: Similarity-aware Knowledge distILLation for Speech Self-Supervised Learningcited by 5
  12. 2025Effective and Efficient Mixed Precision Quantization of Speech Foundation Modelscited by 5
24 more
  1. 2025Efficient Speech Translation through Model Compression and Knowledge Distillationcited by 5
  2. 2024Sustainable self-supervised learning for speech representationscited by 4
  3. 2024One-pass Multiple Conformer and Foundation Speech Systems Compression and Quantization Using An All-in-one Neural Modelcited by 4
  4. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
  5. 2023STaR: Distilling Speech Temporal Relation for Lightweight Speech Self-Supervised Learning Modelscited by 3
  6. 2025DiceHuBERT: Distilling HuBERT with a Self-Supervised Learning Objectivecited by 3
  7. 2025Scaling and Distilling Transformer Models for sEMGcited by 3
  8. 2025Adaptive Knowledge Distillation for Device-Directed Speech Detectioncited by 3
  9. 2024WavRx: a Disease-Agnostic, Generalizable, and Privacy-Preserving Speech Health Diagnostic Modelcited by 2
  10. 2024How Redundant Is the Transformer Stack in Speech Representation Models?cited by 2
  11. 2023Efficiency-oriented approaches for self-supervised speech representation learningcited by 1
  12. 2025Efficient Supernet Training with Orthogonal Softmax for Scalable ASR Model Compressioncited by 1
  13. 2025Effective and Efficient One-pass Compression of Speech Foundation Models Using Sparsity-aware Self-pinching Gatescited by 1
  14. 2025HARNESS: Lightweight Distilled Arabic Speech Foundation Modelscited by 1
  15. 2023GenDistiller: Distilling Pre-trained Language Models based on Generative Modelscited by 0
  16. 2024Active Learning with Task Adaptation Pre-training for Speech Emotion Recognitioncited by 0
  17. 2024AdaPTwin: Low-Cost Adaptive Compression of Product Twins in Transformerscited by 0
  18. 2024GenDistiller: Distilling Pre-trained Language Models based on an Autoregressive Generative Modelcited by 0
  19. 2025Synergistic Effects of Knowledge Distillation and Structured Pruning for Self-Supervised Speech Modelscited by 0
  20. 2026HARNESS: Lightweight Distilled Arabic Speech Foundation Modelscited by 0
  21. 2026Fast Speech Foundation Model Distillation Using Interleaved Stackingcited by 0
  22. 2026Towards Data-free and Training-free Compression for Speech Foundation Models Using Parameter Clusteringcited by 0
  23. 2026Efficiency-Performance Trade-offs in Neural Speaker Diarization via Structured Pruning and Low-Bit Quantizationcited by 0
  24. 2026Task-Aware Joint Pruning and Distillation for Efficient Audio Deepfake Detectioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Hybrid Pruning: In-Situ Compression of Self-Supervised Speech Models for Speaker Verification and Anti-Spoofing
  2. 2023Recycle-and-Distill: Universal Compression Strategy for Transformer-based Speech SSL Models with Attention Map Reusing and Masking Distillation
  3. 2022Is Smaller Always Faster? Tradeoffs in Compressing Self-Supervised Speech Transformers
  4. 2025Fine-tune Before Structured Pruning: Towards Compact and Accurate Self-Supervised Models for Speaker Diarization
  5. 2022FitHuBERT: Going Thinner and Deeper for Knowledge Distillation of Speech Self-Supervised Learning
  6. 2023Accurate and Structured Pruning for Efficient Automatic Speech Recognition
  7. 2023Stable Distillation: Regularizing Continued Pre-training for Low-Resource Automatic Speech Recognition
  8. 2025Distillation and Pruning for Scalable Self-Supervised Representation-Based Speech Quality Assessment