mlsanityData to 14 Oct 2026

Task-Agnostic Structured Pruning of Speech Representation Models

arXiv 2306.01385 ↗First posted 2 Jun 2023eess.AS cs.CL cs.SDcited by 15

Not on the map. Why?

Abstract (arXiv)

Self-supervised pre-trained models such as Wav2vec2, Hubert, and WavLM have been shown to significantly improve many speech tasks. However, their large memory and strong computational requirements hinder their industrial applicability. Structured pruning is a hardware-friendly model compression technique but usually results in a larger loss of accuracy. In this paper, we propose a fine-grained attention head pruning method to compensate for the performance degradation. In addition, we also introduce the straight through estimator into the L0 regularization to further accelerate the pruned model. Experiments on the SUPERB benchmark show that our model can achieve comparable performance to the dense model in multiple tasks and outperforms the Wav2vec 2.0 base model on average, with 72% fewer parameters and 2 times faster inference speed.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 21 of its 25 references that are in the map. First posted on arXiv 2 Jun 2023.

Builds on

21 of 25 references.

References · 21

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  2. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
  3. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
  4. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  5. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
  6. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  7. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  8. 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 666
  9. 2018Rethinking the Value of Network Pruningcited by 599
  10. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 583
  11. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 403
  12. 2022Self-Supervised Speech Representation Learning: A Reviewcited by 260
9 more
  1. 2019Understanding Straight-Through Estimator in Training Activation Quantized Neural Netscited by 231
  2. 2019Structured Pruning of Large Language Modelscited by 203
  3. 2022Structured Pruning Learns Compact and Accurate Modelscited by 170
  4. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 144
  5. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 53
  6. 2022LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERTcited by 50
  7. 2023Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understandingcited by 31
  8. 2022FitHuBERT: Going Thinner and Deeper for Knowledge Distillation of Speech Self-Supervised Learningcited by 25
  9. 2022Learning ASR pathways: A sparse multilingual ASR modelcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 3; 2025: 6; 2026: 3.

15 papers.

Sort:

By area:

  1. 2023CoLLD: Contrastive Layer-to-layer Distillation for Compressing Multilingual Pre-trained Speech Encoderscited by 9
  2. 2024Dynamic Encoder Size Based on Data-Driven Layer-wise Pruning for Speech Recognitioncited by 7
  3. 2025Efficient Speech Translation through Model Compression and Knowledge Distillationcited by 5
  4. 2024DAISY: Data Adaptive Self-Supervised Early Exit for Speech Representation Modelscited by 4
  5. 2023STaR: Distilling Speech Temporal Relation for Lightweight Speech Self-Supervised Learning Modelscited by 3
  6. 2025Speech-FT: Merging Pre-trained And Fine-Tuned Speech Representation Models For Cross-Task Generalizationcited by 2
  7. 2025An Exploration of Mamba for Speech Self-Supervised Modelscited by 2
  8. 2025Efficient Supernet Training with Orthogonal Softmax for Scalable ASR Model Compressioncited by 1
  9. 2025Effective and Efficient One-pass Compression of Speech Foundation Models Using Sparsity-aware Self-pinching Gatescited by 1
  10. 2026Prune Once: Retraining-Free Task-Agnostic Pruning for Vision-Language Modelscited by 1
  11. 2023GenDistiller: Distilling Pre-trained Language Models based on Generative Modelscited by 0
  12. 2024GenDistiller: Distilling Pre-trained Language Models based on an Autoregressive Generative Modelcited by 0
3 more
  1. 2025The silence of the weights: a structural pruning strategy for attention-based audio signal architectures with second order metricscited by 0
  2. 2026Towards Data-free and Training-free Compression for Speech Foundation Models Using Parameter Clusteringcited by 0
  3. 2026Task-Aware Joint Pruning and Distillation for Efficient Audio Deepfake Detectioncited by 0