Sub-8-bit quantization for on-device speech recognition: a regularization-free approach
Not on the map. Why?
Abstract (arXiv)
For on-device automatic speech recognition (ASR), quantization aware training (QAT) is ubiquitous to achieve the trade-off between model predictive performance and efficiency. Among existing QAT methods, one major drawback is that the quantization centroids have to be predetermined and fixed. To overcome this limitation, we introduce a regularization-free, "soft-to-hard" compression mechanism with self-adjustable centroids in a mu-Law constrained space, resulting in a simpler yet more versatile quantization scheme, called General Quantizer (GQ). We apply GQ to ASR tasks using Recurrent Neural Network Transducer (RNN-T) and Conformer architectures on both LibriSpeech and de-identified far-field datasets. Without accuracy degradation, GQ can compress both RNN-T and Conformer into sub-8-bit, and for some RNN-T layers, to 1-bit for fast and accurate inference. We observe a 30.73% memory footprint saving and 31.75% user-perceived latency reduction compared to 8-bit QAT via physical device benchmarking.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 28 of its 37 references that are in the map. First posted on arXiv 17 Oct 2022.
Builds on
28 of 37 references.
References · 28
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
- 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
- 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
- 2021SoundStream: An End-to-End Neural Audio Codeccited by 756
- 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 464
- 2019Q8BERT: Quantized 8Bit BERTcited by 296
- 2021Post-Training Quantization for Vision Transformercited by 274
- 2020TensorFlow Lite Micro: Embedded Machine Learning on TinyML Systemscited by 161
- 2017Soft-to-Hard Vector Quantization for End-to-End Learning Compressible Representationscited by 149
- 2018Exploring Architectures, Data and Units For Streaming End-to-End Speech Recognition with RNN-Transducercited by 95
- 2019Loss Aware Post-training Quantizationcited by 94
16 more
- 2020A Better and Faster End-to-End Model for Streaming ASRcited by 68
- 2019AutoQ: Automated Kernel-Wise Neural Network Quantizationcited by 60
- 2020Permute, Quantize, and Fine-tune: Efficient Compression of Neural Networkscited by 28
- 2022Neural Network Quantization with AI Model Efficiency Toolkit (AIMET)cited by 24
- 20224-bit Conformer with Native Quantization Aware Training for Speech Recognitioncited by 22
- 2021Integer-only Zero-shot Quantization for Efficient Speech Recognitioncited by 19
- 2019Cascaded Cross-Module Residual Learning towards Lightweight End-to-End Speech Codingcited by 15
- 2021Mixed Precision of Quantization of Transformer Language Models for Speech Recognitioncited by 10
- 2021CoDERT: Distilling Encoder Representations with Co-learning for Transducer-based Speech Recognitioncited by 8
- 2020Efficient And Scalable Neural Residual Waveform Coding With Collaborative Quantizationcited by 7
- 2021HARP-Net: Hyper-Autoencoded Reconstruction Propagation for Scalable Neural Audio Codingcited by 7
- 2021Bifocal Neural ASR: Exploiting Keyword Spotting for Inference Optimizationcited by 7
- 20214-bit Quantization of LSTM-based Speech Recognition Modelscited by 7
- 2021Scalable and Efficient Neural Speech Coding: A Hybrid Designcited by 5
- 2022Sub-8-Bit Quantization Aware Training for 8-Bit Neural Network Accelerator with On-Device Speech Recognitioncited by 4
- 2022Cross-Scale Vector Quantization for Scalable Neural Speech Codingcited by 3
Built on by
Citing papers per year: 2024: 1; 2025: 2.
3 papers.