Towards Data-free and Training-free Compression for Speech Foundation Models Using Parameter Clustering
Not on the map. Why?
Abstract (arXiv)
This paper presents a novel data-free and training-free compression approach for speech foundation models using channelwise clustering via k-means. More fine-grained, mixed sparsity pruning by layer-level varying number of parameter clusters is also explored. Experiments conducted on the LibriSpeech dataset suggest that when operating with pruning sparsity of 50% on HuBERT-large, consistent WER reductions of 27.73%/18.61% absolute (34.37%/21.91% relative) over the magnitude-based pruning were obtained on the test-clean and test-other subsets before fine-tuning and 0.19%/0.79% absolute (3.36%/4.62% relative) after fine-tuning with only 3 epochs. Similar WER reductions of 2.86%/5.02% absolute (59.21%/55.29% relative) were observed against magnitudebased pruning on Whisper-large-v3 at 10% sparsity, all with no significant WER increase relative to the uncompressed baseline.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 26 of its 32 references that are in the map. First posted on arXiv 10 Jun 2026.
Builds on
26 of 32 references.
References · 26
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
- 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,905
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
- 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
- 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
- 2021I-BERT: Integer-only BERT Quantizationcited by 255
- 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 144
- 2022LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERTcited by 50
- 2023DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Modelscited by 36
- 2022Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Modelscited by 31
- 2023Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understandingcited by 31
- 2022FitHuBERT: Going Thinner and Deeper for Knowledge Distillation of Speech Self-Supervised Learningcited by 25
14 more
- 20224-bit Conformer with Native Quantization Aware Training for Speech Recognitioncited by 24
- 2023Task-Agnostic Structured Pruning of Speech Representation Modelscited by 15
- 2021Efficient conformer-based speech recognition with linear attentioncited by 13
- 2021Layer Pruning on Demand with Intermediate CTCcited by 13
- 2022PADA: Pruning Assisted Domain Adaptation for Self-Supervised Speech Representationscited by 11
- 2023Accurate and Structured Pruning for Efficient Automatic Speech Recognitioncited by 11
- 20232-bit Conformer quantization for automatic speech recognitioncited by 10
- 2023A Model for Every User and Budget: Label-Free and Personalized Mixed-Precision Quantizationcited by 8
- 2023DistillW2V2: A Small and Streaming Wav2vec 2.0 Based ASR Modelcited by 7
- 2025Effective and Efficient Mixed Precision Quantization of Speech Foundation Modelscited by 5
- 2024One-pass Multiple Conformer and Foundation Speech Systems Compression and Quantization Using An All-in-one Neural Modelcited by 4
- 2025Towards One-bit ASR: Extremely Low-bit Conformer Quantization Using Co-training and Stochastic Precisioncited by 3
- 2025Unfolding A Few Structures for The Many: Memory-Efficient Compression of Conformer and Speech Foundation Modelscited by 1
- 2025Effective and Efficient One-pass Compression of Speech Foundation Models Using Sparsity-aware Self-pinching Gatescited by 1
Built on by
Not cited yet.