Effective and Efficient One-pass Compression of Speech Foundation Models Using Sparsity-aware Self-pinching Gates
Not on the map. Why?
Abstract (arXiv)
This paper presents a novel approach for speech foundation models compression that tightly integrates model pruning and parameter update into a single stage. Highly compact layer-level tied self-pinching gates each containing only a single learnable threshold are jointly trained with uncompressed models and used in fine-grained neuron level pruning. Experiments conducted on the LibriSpeech-100hr corpus suggest that our approach reduces the number of parameters of wav2vec2.0-base and HuBERT-large models by 65% and 60% respectively, while incurring no statistically significant word error rate (WER) increase on the test-clean dataset. Compared to previously published methods on the same task, our approach not only achieves the lowest WER of 7.05% on the test-clean dataset under a comparable model compression ratio of 4.26x, but also operates with at least 25% less model compression time.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 33 of its 41 references that are in the map. First posted on arXiv 28 May 2025.
Builds on
33 of 41 references.
References · 33
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
- 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
- 2018DARTS: Differentiable Architecture Searchcited by 1,782
- 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
- 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
- 2021SUPERB: Speech processing Universal PERformance Benchmarkcited by 652
- 2021I-BERT: Integer-only BERT Quantizationcited by 250
- 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 129
- 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 50
- 2022LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERTcited by 45
- 2023DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Modelscited by 32
21 more
- 2023Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understandingcited by 29
- 2022Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Modelscited by 28
- 2022FitHuBERT: Going Thinner and Deeper for Knowledge Distillation of Speech Self-Supervised Learningcited by 24
- 2020Dynamic Sparsity Neural Networks for Automatic Speech Recognitioncited by 23
- 20224-bit Conformer with Native Quantization Aware Training for Speech Recognitioncited by 22
- 2021Efficient conformer-based speech recognition with linear attentioncited by 13
- 2021Layer Pruning on Demand with Intermediate CTCcited by 13
- 2023Task-Agnostic Structured Pruning of Speech Representation Modelscited by 13
- 2022Learning ASR pathways: A sparse multilingual ASR modelcited by 12
- 2022PADA: Pruning Assisted Domain Adaptation for Self-Supervised Speech Representationscited by 10
- 2023Accurate and Structured Pruning for Efficient Automatic Speech Recognitioncited by 10
- 20232-bit Conformer quantization for automatic speech recognitioncited by 9
- 2023USM-Lite: Quantization and Sparsity Aware Fine-tuning for Speech Recognition with Universal Speech Modelscited by 8
- 2022Losses Can Be Blessings: Routing Self-Supervised Speech Representations Towards Efficient Multilingual and Multitask Speech Processingcited by 7
- 2023DistillW2V2: A Small and Streaming Wav2vec 2.0 Based ASR Modelcited by 7
- 2023A Model for Every User and Budget: Label-Free and Personalized Mixed-Precision Quantizationcited by 7
- 2022Multi-stage Progressive Compression of Conformer Transducer for On-device Speech Recognitioncited by 6
- 2025Effective and Efficient Mixed Precision Quantization of Speech Foundation Modelscited by 5
- 2023Distilling HuBERT with LSTMs via Decoupled Knowledge Distillationcited by 4
- 2024SKILL: Similarity-aware Knowledge distILLation for Speech Self-Supervised Learningcited by 4
- 2024One-pass Multiple Conformer and Foundation Speech Systems Compression and Quantization Using An All-in-one Neural Modelcited by 4
Built on by
Citing papers per year: 2026: 1.
1 paper.