# One Model for All Quantization: A Quantized Network Supporting Hot-Swap Bit-Width Adjustment

arXiv: https://arxiv.org/abs/2105.01353 · HTML: https://mlsanity.com/paper/2105.01353/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 47 references that are in the map.
- First posted on arXiv 4 May 2021.

## Builds on (most cited first, 26 of 26)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Learned Step Size Quantization](https://mlsanity.com/paper/1902.08153/) (2019)
- [DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradients](https://mlsanity.com/paper/1606.06160/) (2016)
- [PACT: Parameterized Clipping Activation for Quantized Neural Networks](https://mlsanity.com/paper/1805.06085/) (2018)
- [Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights](https://mlsanity.com/paper/1702.03044/) (2017)
- [MLPerf Inference Benchmark](https://mlsanity.com/paper/1911.02549/) (2019)
- [Multi-level Wavelet-CNN for Image Restoration](https://mlsanity.com/paper/1805.07071/) (2018)
- [Photorealistic Style Transfer via Wavelet Transforms](https://mlsanity.com/paper/1903.09760/) (2019)
- [Learning to Quantize Deep Networks by Optimizing Quantization Intervals with Task Loss](https://mlsanity.com/paper/1808.05779/) (2018)
- [Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networks](https://mlsanity.com/paper/1909.13144/) (2019)
- [Convolutional Neural Networks using Logarithmic Data Representation](https://mlsanity.com/paper/1603.01025/) (2016)
- [Loss Aware Post-training Quantization](https://mlsanity.com/paper/1911.07190/) (2019)
- [Defensive Quantization: When Efficiency Meets Robustness](https://mlsanity.com/paper/1904.08444/) (2019)
- [Rethinking Differentiable Search for Mixed-Precision Neural Networks](https://mlsanity.com/paper/2004.05795/) (2020)
- [AdaBits: Neural Network Quantization with Adaptive Bit-Widths](https://mlsanity.com/paper/1912.09666/) (2019)
- [Robust Quantization: One Model to Rule Them All](https://mlsanity.com/paper/2002.07686/) (2020)
- [Wavelet Integrated CNNs for Noise-Robust Image Classification](https://mlsanity.com/paper/2005.03337/) (2020)
- [Any-Precision Deep Neural Networks](https://mlsanity.com/paper/1911.07346/) (2019)
- [Attribute-aware Face Aging with Wavelet-based Generative Adversarial Networks](https://mlsanity.com/paper/1809.06647/) (2018)
- [WaveSNet: Wavelet Integrated Deep Networks for Image Segmentation](https://mlsanity.com/paper/2005.14461/) (2020)
- [Effective and Fast: A Novel Sequential Single Path Search for Mixed-Precision Quantization](https://mlsanity.com/paper/2103.02904/) (2021)
- [Multi-Precision Quantized Neural Networks via Encoding Decomposition of -1 and +1](https://mlsanity.com/paper/1905.13389/) (2019)
- [Gradient ℓ1 Regularization for Quantization Robustness](https://mlsanity.com/paper/2002.07520/) (2020)
- [MWQ: Multiscale Wavelet Quantized Neural Networks](https://mlsanity.com/paper/2103.05363/) (2021)

## Built on by (most cited first, 3 of 3)

- [Bitwidth-Adaptive Quantization-Aware Neural Network Training: A Meta-Learning Approach](https://mlsanity.com/paper/2207.10188/) (2022)
- [Quantized Neural Networks via {-1, +1} Encoding Decomposition and Acceleration](https://mlsanity.com/paper/2106.09886/) (2021)
- [Nearly Lossless Adaptive Bit Switching](https://mlsanity.com/paper/2502.01199/) (2025)

## Abstract (arXiv)

As an effective technique to achieve the implementation of deep neural networks in edge devices, model quantization has been successfully applied in many practical applications. No matter the methods of quantization aware training (QAT) or post-training quantization (PTQ), they all depend on the target bit-widths. When the precision of quantization is adjusted, it is necessary to fine-tune the quantized model or minimize the quantization noise, which brings inconvenience in practical applications. In this work, we propose a method to train a model for all quantization that supports diverse bit-widths (e.g., form 8-bit to 1-bit) to satisfy the online quantization bit-width adjustment. It is hot-swappable that can provide specific quantization strategies for different candidates through multiscale quantization. We use wavelet decomposition and reconstruction to increase the diversity of weights, thus significantly improving the performance of each quantization candidate, especially at ultra-low bit-widths (e.g., 3-bit, 2-bit, and 1-bit). Experimental results on ImageNet and COCO show that our method can achieve accuracy comparable performance to dedicated models trained at the same precision.