# 4-bit Conformer with Native Quantization Aware Training for Speech Recognition

arXiv: https://arxiv.org/abs/2203.15952 · HTML: https://mlsanity.com/paper/2203.15952/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 35 references that are in the map.
- First posted on arXiv 29 Mar 2022.

## Builds on (most cited first, 23 of 23)

- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Conformer: Convolution-augmented Transformer for Speech Recognition](https://mlsanity.com/paper/2005.08100/) (2020)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [PACT: Parameterized Clipping Activation for Quantized Neural Networks](https://mlsanity.com/paper/1805.06085/) (2018)
- [Attention-Based Models for Speech Recognition](https://mlsanity.com/paper/1506.07503/) (2015)
- [Deep Speech: Scaling up end-to-end speech recognition](https://mlsanity.com/paper/1412.5567/) (2014)
- [Joint CTC-Attention based End-to-End Speech Recognition using Multi-task Learning](https://mlsanity.com/paper/1609.06773/) (2016)
- [Streaming End-to-end Speech Recognition For Mobile Devices](https://mlsanity.com/paper/1811.06621/) (2018)
- [State-of-the-art Speech Recognition With Sequence-to-Sequence Models](https://mlsanity.com/paper/1712.01769/) (2017)
- [A Streaming On-Device End-to-End Model Surpassing Server-Side Conventional Model Quality and Latency](https://mlsanity.com/paper/2003.12710/) (2020)
- [Lingvo: a Modular and Scalable Framework for Sequence-to-Sequence Modeling](https://mlsanity.com/paper/1902.08295/) (2019)
- [Recognizing long-form speech using streaming end-to-end models](https://mlsanity.com/paper/1910.11455/) (2019)
- [Improving RNN Transducer Modeling for End-to-End Speech Recognition](https://mlsanity.com/paper/1909.12415/) (2019)
- [On the Comparison of Popular End-to-End Models for Large Scale Speech Recognition](https://mlsanity.com/paper/2005.14327/) (2020)
- [Tied & Reduced RNN-T Decoder](https://mlsanity.com/paper/2109.07513/) (2021)
- [A New Training Pipeline for an Improved Neural Transducer](https://mlsanity.com/paper/2005.09319/) (2020)
- [Optimizing Speech Recognition For The Edge](https://mlsanity.com/paper/1909.12408/) (2019)
- [Pareto-Optimal Quantized ResNet Is Mostly 4-bit](https://mlsanity.com/paper/2105.03536/) (2021)
- [Integer-only Zero-shot Quantization for Efficient Speech Recognition](https://mlsanity.com/paper/2103.16827/) (2021)
- [On the efficient representation and execution of deep acoustic models](https://mlsanity.com/paper/1607.04683/) (2016)
- [A Simplified Fully Quantized Transformer for End-to-end Speech Recognition](https://mlsanity.com/paper/1911.03604/) (2019)
- [4-bit Quantization of LSTM-based Speech Recognition Models](https://mlsanity.com/paper/2108.12074/) (2021)
- [Quantization of Acoustic Model Parameters in Automatic Speech Recognition Framework](https://mlsanity.com/paper/2006.09054/) (2020)

## Built on by (most cited first, 24 of 24)

- [A Survey on Model Compression for Large Language Models](https://mlsanity.com/paper/2308.07633/) (2023)
- [Learning ASR pathways: A sparse multilingual ASR model](https://mlsanity.com/paper/2209.05735/) (2022)
- [Q-S5: Towards Quantized State Space Models](https://mlsanity.com/paper/2406.09477/) (2024)
- [A Model for Every User and Budget: Label-Free and Personalized Mixed-Precision Quantization](https://mlsanity.com/paper/2307.12659/) (2023)
- [Online Model Compression for Federated Learning with Large Models](https://mlsanity.com/paper/2205.03494/) (2022)
- [Neural Network Quantization for Microcontrollers: A Comprehensive Survey of Methods, Platforms, and Applications](https://mlsanity.com/paper/2508.15008/) (2025)
- [Effective and Efficient Mixed Precision Quantization of Speech Foundation Models](https://mlsanity.com/paper/2501.03643/) (2025)
- [Sub-8-bit quantization for on-device speech recognition: a regularization-free approach](https://mlsanity.com/paper/2210.09188/) (2022)
- [Sharing Low Rank Conformer Weights for Tiny Always-On Ambient Speech Recognition Models](https://mlsanity.com/paper/2303.08343/) (2023)
- [ResidualTransformer: Residual Low-Rank Learning with Weight-Sharing for Transformer Layers](https://mlsanity.com/paper/2310.02489/) (2023)
- [One-pass Multiple Conformer and Foundation Speech Systems Compression and Quantization Using An All-in-one Neural Model](https://mlsanity.com/paper/2406.10160/) (2024)
- [Outlier Reduction with Gated Attention for Improved Post-training Quantization in Large Sequence-to-sequence Speech Foundation Models](https://mlsanity.com/paper/2406.11022/) (2024)
- [Single Parent Family: A Spectrum of Family Members from a Single Pre-Trained Foundation Model](https://mlsanity.com/paper/2406.19995/) (2024)
- [BitTTS: Highly Compact Text-to-Speech Using 1.58-bit Quantization and Weight Indexing](https://mlsanity.com/paper/2506.03515/) (2025)
- [Enhancing Quantised End-to-End ASR Models via Personalisation](https://mlsanity.com/paper/2309.09136/) (2023)
- [Large Language Model Pruning](https://mlsanity.com/paper/2406.00030/) (2024)
- [Resource-aware Mixed-precision Quantization for Enhancing Deployability of Transformers for Time-series Forecasting on Embedded FPGAs](https://mlsanity.com/paper/2410.03294/) (2024)
- [Effective and Efficient One-pass Compression of Speech Foundation Models Using Sparsity-aware Self-pinching Gates](https://mlsanity.com/paper/2505.22608/) (2025)
- [SEPTQ: A Simple and Effective Post-Training Quantization Paradigm for Large Language Models](https://mlsanity.com/paper/2604.10091/) (2026)
- [RAND: Robustness Aware Norm Decay For Quantized Seq2seq Models](https://mlsanity.com/paper/2305.15536/) (2023)
- [Federated Learning of Large ASR Models in the Real World](https://mlsanity.com/paper/2408.10443/) (2024)
- [Diagnostic-Driven Layer-Wise Compensation for Post-Training Quantization of Encoder-Decoder ASR Models](https://mlsanity.com/paper/2601.02455/) (2026)
- [Towards Data-free and Training-free Compression for Speech Foundation Models Using Parameter Clustering](https://mlsanity.com/paper/2606.11836/) (2026)
- [O'ido: Open-Vocabulary Speech Recognition on a 5 Microcontroller without a Neural Accelerator](https://mlsanity.com/paper/2610.13447/) (2026)

## Abstract (arXiv)

Reducing the latency and model size has always been a significant research problem for live Automatic Speech Recognition (ASR) application scenarios. Along this direction, model quantization has become an increasingly popular approach to compress neural networks and reduce computation cost. Most of the existing practical ASR systems apply post-training 8-bit quantization. To achieve a higher compression rate without introducing additional performance regression, in this study, we propose to develop 4-bit ASR models with native quantization aware training, which leverages native integer operations to effectively optimize both training and inference. We conducted two experiments on state-of-the-art Conformer-based ASR models to evaluate our proposed quantization technique. First, we explored the impact of different precisions for both weight and activation quantization on the LibriSpeech dataset, and obtained a lossless 4-bit Conformer model with 5.8x size reduction compared to the float32 model. Following this, we for the first time investigated and revealed the viability of 4-bit quantization on a practical ASR system that is trained with large-scale datasets, and produced a lossless Conformer ASR model with mixed 4-bit and 8-bit weights that has 5x size reduction compared to the float32 model.