# Effective and Efficient Mixed Precision Quantization of Speech Foundation Models arXiv: https://arxiv.org/abs/2501.03643 ยท HTML: https://mlsanity.com/paper/2501.03643/ ## Key facts - Cited by 5 papers in the map (data through 14 Oct 2026). - Builds on 37 of its 45 references that are in the map. - First posted on arXiv 7 Jan 2025. ## Builds on (most cited first, 30 of 37) - [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020) - [HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units](https://mlsanity.com/paper/2106.07447/) (2021) - [DARTS: Differentiable Architecture Search](https://mlsanity.com/paper/1806.09055/) (2018) - [WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing](https://mlsanity.com/paper/2110.13900/) (2021) - [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016) - [ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformers](https://mlsanity.com/paper/2206.01861/) (2022) - [A White Paper on Neural Network Quantization](https://mlsanity.com/paper/2106.08295/) (2021) - [ZeroQ: A Novel Zero Shot Quantization Framework](https://mlsanity.com/paper/2001.00281/) (2020) - [I-BERT: Integer-only BERT Quantization](https://mlsanity.com/paper/2101.01321/) (2021) - [DistilHuBERT: Speech Representation Learning by Layer-wise Distillation of Hidden-unit BERT](https://mlsanity.com/paper/2110.01900/) (2021) - [DSNAS: Direct Neural Architecture Search without Parameter Retraining](https://mlsanity.com/paper/2002.09128/) (2020) - [PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognition](https://mlsanity.com/paper/2106.05933/) (2021) - [LightHuBERT: Lightweight and Configurable Speech Representation Learning with Once-for-All Hidden-Unit BERT](https://mlsanity.com/paper/2203.15610/) (2022) - [DPHuBERT: Joint Distillation and Pruning of Self-Supervised Speech Models](https://mlsanity.com/paper/2305.17651/) (2023) - [Deep versus Wide: An Analysis of Student Architectures for Task-Agnostic Knowledge Distillation of Self-Supervised Speech Models](https://mlsanity.com/paper/2207.06867/) (2022) - [Structured Pruning of Self-Supervised Pre-trained Models for Speech Recognition and Understanding](https://mlsanity.com/paper/2302.14132/) (2023) - [FitHuBERT: Going Thinner and Deeper for Knowledge Distillation of Speech Self-Supervised Learning](https://mlsanity.com/paper/2207.00555/) (2022) - [4-bit Conformer with Native Quantization Aware Training for Speech Recognition](https://mlsanity.com/paper/2203.15952/) (2022) - [Dynamic Sparsity Neural Networks for Automatic Speech Recognition](https://mlsanity.com/paper/2005.10627/) (2020) - [Shrinking Bigfoot: Reducing wav2vec 2.0 footprint](https://mlsanity.com/paper/2103.15760/) (2021) - [Integer-only Zero-shot Quantization for Efficient Speech Recognition](https://mlsanity.com/paper/2103.16827/) (2021) - [Automatic Mixed-Precision Quantization Search of BERT](https://mlsanity.com/paper/2112.14938/) (2021) - [Efficient conformer-based speech recognition with linear attention](https://mlsanity.com/paper/2104.06865/) (2021) - [Layer Pruning on Demand with Intermediate CTC](https://mlsanity.com/paper/2106.09216/) (2021) - [PADA: Pruning Assisted Domain Adaptation for Self-Supervised Speech Representations](https://mlsanity.com/paper/2203.16965/) (2022) - [Accurate and Structured Pruning for Efficient Automatic Speech Recognition](https://mlsanity.com/paper/2305.19549/) (2023) - [2-bit Conformer quantization for automatic speech recognition](https://mlsanity.com/paper/2305.16619/) (2023) - [Multi-stage Progressive Compression of Conformer Transducer for On-device Speech Recognition](https://mlsanity.com/paper/2210.00169/) (2022) - [CoLLD: Contrastive Layer-to-layer Distillation for Compressing Multilingual Pre-trained Speech Encoders](https://mlsanity.com/paper/2309.07707/) (2023) - [USM-Lite: Quantization and Sparsity Aware Fine-tuning for Speech Recognition with Universal Speech Models](https://mlsanity.com/paper/2312.08553/) (2023) ## Built on by (most cited first, 5 of 5) - [Effective and Efficient One-pass Compression of Speech Foundation Models Using Sparsity-aware Self-pinching Gates](https://mlsanity.com/paper/2505.22608/) (2025) - [Diagnostic-Driven Layer-Wise Compensation for Post-Training Quantization of Encoder-Decoder ASR Models](https://mlsanity.com/paper/2601.02455/) (2026) - [Mixed-Precision Information Bottlenecks for On-Device Trait-State Disentanglement in Bipolar Agitation Detection](https://mlsanity.com/paper/2605.03039/) (2026) - [Towards Data-free and Training-free Compression for Speech Foundation Models Using Parameter Clustering](https://mlsanity.com/paper/2606.11836/) (2026) - [Scaling Audio Models Efficiently: Joint Optimization of Scale, Resolution, Adaptation, Precision, and Sparsity](https://mlsanity.com/paper/2606.22790/) (2026) ## Abstract (arXiv) This paper presents a novel mixed-precision quantization approach for speech foundation models that tightly integrates mixed-precision learning and quantized model parameter estimation into one single model compression stage. Experiments conducted on LibriSpeech dataset with fine-tuned wav2vec2.0-base and HuBERT-large models suggest the resulting mixed-precision quantized models increased the lossless compression ratio by factors up to 1.7x and 1.9x over the respective uniform-precision and two-stage mixed-precision quantized baselines that perform precision learning and model parameters quantization in separate and disjointed stages, while incurring no statistically word error rate (WER) increase over the 32-bit full-precision models. The system compression time of wav2vec2.0-base and HuBERT-large models is reduced by up to 1.9 and 1.5 times over the two-stage mixed-precision baselines, while both produce lower WERs. The best-performing 3.5-bit mixed-precision quantized HuBERT-large model produces a lossless compression ratio of 8.6x over the 32-bit full-precision system.