# Integer-only Zero-shot Quantization for Efficient Speech Recognition

arXiv: https://arxiv.org/abs/2103.16827 · HTML: https://mlsanity.com/paper/2103.16827/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 29 references that are in the map.
- First posted on arXiv 31 Mar 2021.

## Builds on (most cited first, 23 of 23)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT](https://mlsanity.com/paper/1909.05840/) (2019)
- [Q8BERT: Quantized 8Bit BERT](https://mlsanity.com/paper/1910.06188/) (2019)
- [ZeroQ: A Novel Zero Shot Quantization Framework](https://mlsanity.com/paper/2001.00281/) (2020)
- [I-BERT: Integer-only BERT Quantization](https://mlsanity.com/paper/2101.01321/) (2021)
- [Transformer Transducer: A Streamable Speech Recognition Model with Transformer Encoders and RNN-T Loss](https://mlsanity.com/paper/2002.02562/) (2020)
- [Data-Free Learning of Student Networks](https://mlsanity.com/paper/1904.01186/) (2019)
- [ContextNet: Improving Convolutional Neural Networks for Automatic Speech Recognition with Global Context](https://mlsanity.com/paper/2005.03191/) (2020)
- [QuartzNet: Deep Automatic Speech Recognition with 1D Time-Channel Separable Convolutions](https://mlsanity.com/paper/1910.10261/) (2019)
- [Bayesian Bits: Unifying Quantization and Pruning](https://mlsanity.com/paper/2005.07093/) (2020)
- [Generative Low-bitwidth Data Free Quantization](https://mlsanity.com/paper/2003.03603/) (2020)
- [Data-Free Network Quantization With Adversarial Knowledge Distillation](https://mlsanity.com/paper/2005.04136/) (2020)
- [Jasper: An End-to-End Convolutional Neural Acoustic Model](https://mlsanity.com/paper/1904.03288/) (2019)
- [HAWQV3: Dyadic Neural Network Quantization](https://mlsanity.com/paper/2011.10680/) (2020)
- [The Knowledge Within: Methods for Data-Free Model Compression](https://mlsanity.com/paper/1912.01274/) (2019)
- [Generative Zero-shot Network Quantization](https://mlsanity.com/paper/2101.08430/) (2021)
- [A Simplified Fully Quantized Transformer for End-to-end Speech Recognition](https://mlsanity.com/paper/1911.03604/) (2019)
- [INT8 Winograd Acceleration for Conv1D Equipped ASR Models Deployed on Mobile Devices](https://mlsanity.com/paper/2010.14841/) (2020)
- [Quantization of Acoustic Model Parameters in Automatic Speech Recognition Framework](https://mlsanity.com/paper/2006.09054/) (2020)

## Built on by (most cited first, 19 of 19)

- [Full Stack Optimization of Transformer Inference: a Survey](https://mlsanity.com/paper/2302.14017/) (2023)
- [4-bit Conformer with Native Quantization Aware Training for Speech Recognition](https://mlsanity.com/paper/2203.15952/) (2022)
- [A Model for Every User and Budget: Label-Free and Personalized Mixed-Precision Quantization](https://mlsanity.com/paper/2307.12659/) (2023)
- [Efficient Deployment of Spiking Neural Networks on SpiNNaker2 for DVS Gesture Recognition Using Neuromorphic Intermediate Representation](https://mlsanity.com/paper/2504.06748/) (2025)
- [Effective and Efficient Mixed Precision Quantization of Speech Foundation Models](https://mlsanity.com/paper/2501.03643/) (2025)
- [Sub-8-bit quantization for on-device speech recognition: a regularization-free approach](https://mlsanity.com/paper/2210.09188/) (2022)
- [One-pass Multiple Conformer and Foundation Speech Systems Compression and Quantization Using An All-in-one Neural Model](https://mlsanity.com/paper/2406.10160/) (2024)
- [Accelerating Inference and Language Model Fusion of Recurrent Neural Network Transducers via End-to-End 4-bit Quantization](https://mlsanity.com/paper/2206.07882/) (2022)
- [Quantization for OpenAI's Whisper Models: A Comparative Analysis](https://mlsanity.com/paper/2503.09905/) (2025)
- [DilateQuant: Accurate and Efficient Diffusion Quantization via Weight Dilation](https://mlsanity.com/paper/2409.14307/) (2024)
- [Rethinking Small VLM Quantization: From Component-Wise Analysis to Hardware-Aware Edge Deployment](https://mlsanity.com/paper/2607.08029/) (2026)
- [Oracle Teacher: Leveraging Target Information for Better Knowledge Distillation of CTC Models](https://mlsanity.com/paper/2111.03664/) (2021)
- [Deep learning model compression using network sensitivity and gradients](https://mlsanity.com/paper/2210.05111/) (2022)
- [RAND: Robustness Aware Norm Decay For Quantized Seq2seq Models](https://mlsanity.com/paper/2305.15536/) (2023)
- [Quantizing Whisper-small: How design choices affect ASR performance](https://mlsanity.com/paper/2511.08093/) (2025)
- [Parallax: Runtime Parallelization for Operator Fallbacks in Heterogeneous Edge Systems](https://mlsanity.com/paper/2512.11532/) (2025)
- [Diagnostic-Driven Layer-Wise Compensation for Post-Training Quantization of Encoder-Decoder ASR Models](https://mlsanity.com/paper/2601.02455/) (2026)
- [TARQ: Tail-Aware Reconstruction Quantization for Rare-Word Robust Automatic Speech Recognition](https://mlsanity.com/paper/2605.27808/) (2026)
- [I-Parakeet: Integer-Only Conformer ASR on Mobile NPU](https://mlsanity.com/paper/2609.30846/) (2026)

## Abstract (arXiv)

End-to-end neural network models achieve improved performance on various automatic speech recognition (ASR) tasks. However, these models perform poorly on edge hardware due to large memory and computation requirements. While quantizing model weights and/or activations to low-precision can be a promising solution, previous research on quantizing ASR models is limited. In particular, the previous approaches use floating-point arithmetic during inference and thus they cannot fully exploit efficient integer processing units. Moreover, they require training and/or validation data during quantization, which may not be available due to security or privacy concerns. To address these limitations, we propose an integer-only, zero-shot quantization scheme for ASR models. In particular, we generate synthetic data whose runtime statistics resemble the real data, and we use it to calibrate models during quantization. We apply our method to quantize QuartzNet, Jasper, and Conformer and show negligible WER degradation as compared to the full-precision baseline models, even without using any data. Moreover, we achieve up to 2.35x speedup on a T4 GPU and 4x compression rate, with a modest WER degradation of <1% with INT8 quantization.