mlsanityData to 7 Oct 2026

Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understanding

arXiv 2302.14132 ↗First posted 27 Feb 2023cs.CL cs.SD eess.AScited by 29

Not on the map. Why?

Abstract (arXiv)

Self-supervised speech representation learning (SSL) has shown to be effective in various downstream tasks, but SSL models are usually large and slow. Model compression techniques such as pruning aim to reduce the model size and computation without degradation in accuracy. Prior studies focus on the pruning of Transformers; however, speech models not only utilize a stack of Transformer blocks, but also combine a frontend network based on multiple convolutional layers for low-level feature representation learning. This frontend has a small size but a heavy computational cost. In this work, we propose three task-specific structured pruning methods to deal with such heterogeneous networks. Experiments on LibriSpeech and SLURP show that the proposed method is more accurate than the original wav2vec2-base with 10% to 30% less computation, and is able to reduce the computation by 40% to 50% without any degradation.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Builds on 37 of its 45 references that are in the map. First posted on arXiv 27 Feb 2023.

Builds on

37 of 45 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  5. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  6. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  7. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  8. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  9. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  10. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  11. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  12. 2016Pruning Filters for Efficient ConvNetscited by 1,129
25 more
  1. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 673
  2. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
  3. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
  4. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
  5. 2019Structured Pruning of Large Language Modelscited by 201
  6. 2020SLURP: A Spoken Language Understanding Resource Packagecited by 197
  7. 2019Speech Model Pre-training for End-to-End Spoken Language Understandingcited by 191
  8. 2022Structured Pruning Learns Compact and Accurate Modelscited by 167
  9. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 129
  10. 2022Branchformer: Parallel MLP-Attention Architectures to Capture Local and Global Context for Speech Recognition and Understandingcited by 90
  11. 2016Group Sparse Regularization for Deep Neural Networkscited by 89
  12. 2022PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importancecited by 79
  13. 2021SLUE: New Benchmark Tasks for Spoken Language Understanding Evaluation on Natural Speechcited by 70
  14. 2021Performance-Efficiency Trade-offs in Unsupervised Pre-training for Speech Recognitioncited by 68
  15. 2021Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalizationcited by 56
  16. 2021ESPnet-SLU: Advancing Spoken Language Understanding through ESPnetcited by 53
  17. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 50
  18. 2022LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERTcited by 45
  19. 2021An Exploration of Self-Supervised Pretrained Representations for End-to-End Speech Recognitioncited by 39
  20. 2022Investigating self-supervised learning for speech enhancement and separationcited by 35
  21. 2022Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo Languagescited by 26
  22. 2020On the Usefulness of Self-Attention for Automatic Speech Recognition with Transformerscited by 24
  23. 2020RTMobile: Beyond Real-Time Mobile Acceleration of RNNs for Speech Recognitioncited by 19
  24. 2022A Study on the Integration of Pre-trained SSL, ASR, LM and SLU Models for Spoken Language Understandingcited by 18
  25. 2021Shrinking Bigfoot: Reducing wav2vec 2.0 footprintcited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 12; 2025: 7; 2026: 3.

29 papers.

Sort:

By area:

  1. 2024OWSM v3.1: Better and Faster Open Whisper-Style Speech Models based on E-Branchformercited by 74
  2. 2023Reproducing Whisper-Style Training Using an Open-Source Toolkit and Publicly Available Datacited by 42
  3. 2024OWSM-CTC: An Open Encoder-Only Speech Foundation Model for Speech Recognition, Translation, and Language Identificationcited by 34
  4. 2023DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Modelscited by 32
  5. 2023Reducing Barriers to Self-Supervised Learning: HuBERT Pre-training with Academic Computecited by 21
  6. 2023Task-Agnostic Structured Pruning of Speech Representation Modelscited by 13
  7. 2023DQ-Whisper: Joint Distillation and Quantization for Efficient Multilingual Speech Recognitioncited by 10
  8. 2023CoLLD: Contrastive Layer-to-layer Distillation for Compressing Multilingual Pre-trained Speech Encoderscited by 7
  9. 2024Dynamic Encoder Size Based on Data-Driven Layer-wise Pruning for Speech Recognitioncited by 7
  10. 2024Efficient Training of Self-Supervised Speech Foundation Models on a Compute Budgetcited by 5
  11. 2025Effective and Efficient Mixed Precision Quantization of Speech Foundation Modelscited by 5
  12. 2024SKILL: Similarity-aware Knowledge distILLation for Speech Self-Supervised Learningcited by 4
17 more
  1. 2024DAISY: Data Adaptive Self-Supervised Early Exit for Speech Representation Modelscited by 4
  2. 2024Sustainable self-supervised learning for speech representationscited by 4
  3. 2024One-pass Multiple Conformer and Foundation Speech Systems Compression and Quantization Using An All-in-one Neural Modelcited by 4
  4. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
  5. 2025On-device Streaming Discrete Speech Unitscited by 4
  6. 2025StableQuant: Layer Adaptive Post-Training Quantization for Speech Foundation Modelscited by 3
  7. 2024Finding Task-specific Subnetworks in Multi-task Spoken Language Understanding Modelcited by 2
  8. 2024Convexity-based Pruning of Speech Representation Modelscited by 2
  9. 2024How Redundant Is the Transformer Stack in Speech Representation Models?cited by 2
  10. 2025Pruning as Regularization: Sensitivity-Aware One-Shot Pruning in ASRcited by 2
  11. 2023Efficiency-oriented approaches for self-supervised speech representation learningcited by 1
  12. 2025Efficient Supernet Training with Orthogonal Softmax for Scalable ASR Model Compressioncited by 1
  13. 2025Effective and Efficient One-pass Compression of Speech Foundation Models Using Sparsity-aware Self-pinching Gatescited by 1
  14. 2025The silence of the weights: a structural pruning strategy for attention-based audio signal architectures with second order metricscited by 0
  15. 2026Measuring the Redundancy of Decoder Layers in SpeechLLMscited by 0
  16. 2026Towards Data-free and Training-free Compression for Speech Foundation Models Using Parameter Clusteringcited by 0
  17. 2026Reduced Matrix Multiplication: Input-Adaptive Matrix-Product Reduction for LLM Inferencecited by 0