# Sub-8-bit quantization for on-device speech recognition: a regularization-free approach

arXiv: https://arxiv.org/abs/2210.09188 · HTML: https://mlsanity.com/paper/2210.09188/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 37 references that are in the map.
- First posted on arXiv 17 Oct 2022.

## Builds on (most cited first, 28 of 28)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognition](https://mlsanity.com/paper/1904.08779/) (2019)
- [SoundStream: An End-to-End Neural Audio Codec](https://mlsanity.com/paper/2107.03312/) (2021)
- [BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstruction](https://mlsanity.com/paper/2102.05426/) (2021)
- [Q8BERT: Quantized 8Bit BERT](https://mlsanity.com/paper/1910.06188/) (2019)
- [Post-Training Quantization for Vision Transformer](https://mlsanity.com/paper/2106.14156/) (2021)
- [TensorFlow Lite Micro: Embedded Machine Learning on TinyML Systems](https://mlsanity.com/paper/2010.08678/) (2020)
- [Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representations](https://mlsanity.com/paper/1704.00648/) (2017)
- [Loss Aware Post-training Quantization](https://mlsanity.com/paper/1911.07190/) (2019)
- [Exploring Architectures, Data and Units For Streaming End-to-End Speech Recognition with RNN-Transducer](https://mlsanity.com/paper/1801.00841/) (2018)
- [A Better and Faster End-to-End Model for Streaming ASR](https://mlsanity.com/paper/2011.10798/) (2020)
- [AutoQ: Automated Kernel-Wise Neural Network Quantization](https://mlsanity.com/paper/1902.05690/) (2019)
- [Permute, Quantize, and Fine-tune: Efficient Compression of Neural Networks](https://mlsanity.com/paper/2010.15703/) (2020)
- [Neural Network Quantization with AI Model Efficiency Toolkit (AIMET)](https://mlsanity.com/paper/2201.08442/) (2022)
- [4-bit Conformer with Native Quantization Aware Training for Speech Recognition](https://mlsanity.com/paper/2203.15952/) (2022)
- [Integer-only Zero-shot Quantization for Efficient Speech Recognition](https://mlsanity.com/paper/2103.16827/) (2021)
- [Cascaded Cross-Module Residual Learning towards Lightweight End-to-End Speech Coding](https://mlsanity.com/paper/1906.07769/) (2019)
- [CoDERT: Distilling Encoder Representations with Co-learning for Transducer-based Speech Recognition](https://mlsanity.com/paper/2106.07734/) (2021)
- [Mixed Precision of Quantization of Transformer Language Models for Speech Recognition](https://mlsanity.com/paper/2112.11540/) (2021)
- [4-bit Quantization of LSTM-based Speech Recognition Models](https://mlsanity.com/paper/2108.12074/) (2021)
- [Efficient And Scalable Neural Residual Waveform Coding With Collaborative Quantization](https://mlsanity.com/paper/2002.05604/) (2020)
- [HARP-Net: Hyper-Autoencoded Reconstruction Propagation for Scalable Neural Audio Coding](https://mlsanity.com/paper/2107.10843/) (2021)
- [Bifocal Neural ASR: Exploiting Keyword Spotting for Inference Optimization](https://mlsanity.com/paper/2108.01704/) (2021)
- [Scalable and Efficient Neural Speech Coding: A Hybrid Design](https://mlsanity.com/paper/2103.14776/) (2021)
- [Sub-8-Bit Quantization Aware Training for 8-Bit Neural Network Accelerator with On-Device Speech Recognition](https://mlsanity.com/paper/2206.15408/) (2022)
- [Cross-Scale Vector Quantization for Scalable Neural Speech Coding](https://mlsanity.com/paper/2207.03067/) (2022)

## Built on by (most cited first, 4 of 4)

- [Effective and Efficient Mixed Precision Quantization of Speech Foundation Models](https://mlsanity.com/paper/2501.03643/) (2025)
- [Quantization for OpenAI's Whisper Models: A Comparative Analysis](https://mlsanity.com/paper/2503.09905/) (2025)
- [Federated Learning of Large ASR Models in the Real World](https://mlsanity.com/paper/2408.10443/) (2024)
- [O'ido: Open-Vocabulary Speech Recognition on a 5 Microcontroller without a Neural Accelerator](https://mlsanity.com/paper/2610.13447/) (2026)

## Abstract (arXiv)

For on-device automatic speech recognition (ASR), quantization aware training (QAT) is ubiquitous to achieve the trade-off between model predictive performance and efficiency. Among existing QAT methods, one major drawback is that the quantization centroids have to be predetermined and fixed. To overcome this limitation, we introduce a regularization-free, "soft-to-hard" compression mechanism with self-adjustable centroids in a mu-Law constrained space, resulting in a simpler yet more versatile quantization scheme, called General Quantizer (GQ). We apply GQ to ASR tasks using Recurrent Neural Network Transducer (RNN-T) and Conformer architectures on both LibriSpeech and de-identified far-field datasets. Without accuracy degradation, GQ can compress both RNN-T and Conformer into sub-8-bit, and for some RNN-T layers, to 1-bit for fast and accurate inference. We observe a 30.73% memory footprint saving and 31.75% user-perceived latency reduction compared to 8-bit QAT via physical device benchmarking.