mlsanityData to 7 Oct 2026

Effective and Efficient Mixed Precision Quantization of Speech Foundation Models

arXiv 2501.03643 ↗First posted 7 Jan 2025cs.SD cs.AI eess.AScited by 5

Not on the map. Why?

Abstract (arXiv)

This paper presents a novel mixed-precision quantization approach for speech foundation models that tightly integrates mixed-precision learning and quantized model parameter estimation into one single model compression stage. Experiments conducted on LibriSpeech dataset with fine-tuned wav2vec2.0-base and HuBERT-large models suggest the resulting mixed-precision quantized models increased the lossless compression ratio by factors up to 1.7x and 1.9x over the respective uniform-precision and two-stage mixed-precision quantized baselines that perform precision learning and model parameters quantization in separate and disjointed stages, while incurring no statistically word error rate (WER) increase over the 32-bit full-precision models. The system compression time of wav2vec2.0-base and HuBERT-large models is reduced by up to 1.9 and 1.5 times over the two-stage mixed-precision baselines, while both produce lower WERs. The best-performing 3.5-bit mixed-precision quantized HuBERT-large model produces a lossless compression ratio of 8.6x over the 32-bit full-precision system.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 37 of its 45 references that are in the map. First posted on arXiv 7 Jan 2025.

Builds on

37 of 45 references.

References · 37

  1. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  2. 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
  3. 2018DARTS: Differentiable Architecture Searchcited by 1,782
  4. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,609
  5. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  6. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  7. 2021A White Paper on Neural Network Quantizationcited by 453
  8. 2020ZeroQ: A Novel Zero Shot Quantization Frameworkcited by 265
  9. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  10. 2021DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERTcited by 129
  11. 2020DSNAS: Direct Neural Architecture Search without Parameter Retrainingcited by 69
  12. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 50
25 more
  1. 2022LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERTcited by 45
  2. 2023DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Modelscited by 32
  3. 2023Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understandingcited by 29
  4. 2022Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Modelscited by 28
  5. 2022FitHuBERT: Going Thinner and Deeper for Knowledge Distillation of Speech Self-Supervised Learningcited by 24
  6. 2020Dynamic Sparsity Neural Networks for Automatic Speech Recognitioncited by 23
  7. 20224-bit Conformer with Native Quantization Aware Training for Speech Recognitioncited by 22
  8. 2021Integer-only Zero-shot Quantization for Efficient Speech Recognitioncited by 19
  9. 2021Shrinking Bigfoot: Reducing wav2vec 2.0 footprintcited by 16
  10. 2021Automatic Mixed-Precision Quantization Search of BERTcited by 16
  11. 2021Efficient conformer-based speech recognition with linear attentioncited by 13
  12. 2021Layer Pruning on Demand with Intermediate CTCcited by 13
  13. 2022PADA: Pruning Assisted Domain Adaptation for Self-Supervised Speech Representationscited by 10
  14. 2023Accurate and Structured Pruning for Efficient Automatic Speech Recognitioncited by 10
  15. 20232-bit Conformer quantization for automatic speech recognitioncited by 9
  16. 2023USM-Lite: Quantization and Sparsity Aware Fine-tuning for Speech Recognition with Universal Speech Modelscited by 8
  17. 2023DistillW2V2: A Small and Streaming Wav2vec 2.0 Based ASR Modelcited by 7
  18. 2023A Model for Every User and Budget: Label-Free and Personalized Mixed-Precision Quantizationcited by 7
  19. 2023CoLLD: Contrastive Layer-to-layer Distillation for Compressing Multilingual Pre-trained Speech Encoderscited by 7
  20. 2022Multi-stage Progressive Compression of Conformer Transducer for On-device Speech Recognitioncited by 6
  21. 2021Mixed Precision Low-bit Quantization of Neural Network Language Models for Speech Recognitioncited by 5
  22. 2023Distilling HuBERT with LSTMs via Decoupled Knowledge Distillationcited by 4
  23. 2024One-pass Multiple Conformer and Foundation Speech Systems Compression and Quantization Using An All-in-one Neural Modelcited by 4
  24. 2022Sub-8-bit quantization for on-device speech recognition: a regularization-free approachcited by 3
  25. 2023Enhancing Quantised End-to-End ASR Models via Personalisationcited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 4.

5 papers.

  1. 2025Effective and Efficient One-pass Compression of Speech Foundation Models Using Sparsity-aware Self-pinching Gatescited by 1
  2. 2026Diagnostic-Driven Layer-Wise Compensation for Post-Training Quantization of Encoder-Decoder ASR Modelscited by 0
  3. 2026Mixed-Precision Information Bottlenecks for On-Device Trait-State Disentanglement in Bipolar Agitation Detectioncited by 0
  4. 2026Towards Data-free and Training-free Compression for Speech Foundation Models Using Parameter Clusteringcited by 0
  5. 2026Scaling Audio Models Efficiently: Joint Optimization of Scale, Resolution, Adaptation, Precision, and Sparsitycited by 0