# Quantization-Guided Training for Compact TinyML Models

arXiv: https://arxiv.org/abs/2103.06231 · HTML: https://mlsanity.com/paper/2103.06231/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 37 references that are in the map.
- First posted on arXiv 10 Mar 2021.

## Builds on (most cited first, 23 of 23)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks](https://mlsanity.com/paper/1803.03635/) (2018)
- [Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference](https://mlsanity.com/paper/1712.05877/) (2017)
- [ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices](https://mlsanity.com/paper/1707.01083/) (2017)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [Once-for-All: Train One Network and Specialize it for Efficient Deployment](https://mlsanity.com/paper/1908.09791/) (2019)
- [Learning Sparse Neural Networks through L0 Regularization](https://mlsanity.com/paper/1712.01312/) (2017)
- [Quantizing deep convolutional networks for efficient inference: A whitepaper](https://mlsanity.com/paper/1806.08342/) (2018)
- [A systematic study of the class imbalance problem in convolutional neural networks](https://mlsanity.com/paper/1710.05381/) (2017)
- [Training with Quantization Noise for Extreme Model Compression](https://mlsanity.com/paper/2004.07320/) (2020)
- [Well-Read Students Learn Better: On the Importance of Pre-training Compact Models](https://mlsanity.com/paper/1908.08962/) (2019)
- [Train Large, Then Compress: Rethinking Model Size for Efficient Training and Inference of Transformers](https://mlsanity.com/paper/2002.11794/) (2020)
- [Visual Wake Words Dataset](https://mlsanity.com/paper/1906.05721/) (2019)
- [Discovering Low-Precision Networks Close to Full-Precision Networks for Efficient Embedded Inference](https://mlsanity.com/paper/1809.04191/) (2018)
- [Bit Efficient Quantization for Deep Neural Networks](https://mlsanity.com/paper/1910.04877/) (2019)
- [Generalized Ternary Connect: End-to-End Learning and Compression of Multiplication-Free Deep Neural Networks](https://mlsanity.com/paper/1811.04985/) (2018)
- [SinReQ: Generalized Sinusoidal Regularization for Low-Bitwidth Deep Quantized Training](https://mlsanity.com/paper/1905.01416/) (2019)
- [Subtensor Quantization for Mobilenets](https://mlsanity.com/paper/2011.08009/) (2020)

## Built on by (most cited first, 10 of 10)

- [Implicit Neural Representations for Image Compression](https://mlsanity.com/paper/2112.04267/) (2021)
- [Overcoming Oscillations in Quantization-Aware Training](https://mlsanity.com/paper/2203.11086/) (2022)
- [A TinyML Platform for On-Device Continual Learning with Quantized Latent Replays](https://mlsanity.com/paper/2110.10486/) (2021)
- [An Empirical Study of Low Precision Quantization for TinyML](https://mlsanity.com/paper/2203.05492/) (2022)
- [From Tiny Machine Learning to Tiny Deep Learning: A Survey](https://mlsanity.com/paper/2506.18927/) (2025)
- [Quantizing Small-Scale State-Space Models for Edge AI](https://mlsanity.com/paper/2506.12480/) (2025)
- [LightDepth: A Resource Efficient Depth Estimation Approach for Dealing with Ground Truth Sparsity via Curriculum Learning](https://mlsanity.com/paper/2211.08608/) (2022)
- [Extreme Compression of Adaptive Neural Images](https://mlsanity.com/paper/2405.16807/) (2024)
- [Deep learning model compression using network sensitivity and gradients](https://mlsanity.com/paper/2210.05111/) (2022)
- [Toward Efficient Convolutional Neural Networks With Structured Ternary Patterns](https://mlsanity.com/paper/2407.14831/) (2024)

## Abstract (arXiv)

We propose a Quantization Guided Training (QGT) method to guide DNN training towards optimized low-bit-precision targets and reach extreme compression levels below 8-bit precision. Unlike standard quantization-aware training (QAT) approaches, QGT uses customized regularization to encourage weight values towards a distribution that maximizes accuracy while reducing quantization errors. One of the main benefits of this approach is the ability to identify compression bottlenecks. We validate QGT using state-of-the-art model architectures on vision datasets. We also demonstrate the effectiveness of QGT with an 81KB tiny model for person detection down to 2-bit precision (representing 17.7x size reduction), while maintaining an accuracy drop of only 3% compared to a floating-point baseline.