# One Weight Bitwidth to Rule Them All

arXiv: https://arxiv.org/abs/2008.09916 · HTML: https://mlsanity.com/paper/2008.09916/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Aug 2020.

## Built on by (most cited first, 13 of 13)

- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [A White Paper on Neural Network Quantization](https://mlsanity.com/paper/2106.08295/) (2021)
- [A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classification](https://mlsanity.com/paper/2205.07877/) (2022)
- [Patch Similarity Aware Data-Free Quantization for Vision Transformers](https://mlsanity.com/paper/2203.02250/) (2022)
- [PSAQ-ViT V2: Towards Accurate and General Data-Free Quantization for Vision Transformers](https://mlsanity.com/paper/2209.05687/) (2022)
- [OMPQ: Orthogonal Mixed Precision Quantization](https://mlsanity.com/paper/2109.07865/) (2021)
- [Applications and Techniques for Fast Machine Learning in Science](https://mlsanity.com/paper/2110.13041/) (2021)
- [EMQ: Evolving Training-free Proxies for Automated Mixed Precision Quantization](https://mlsanity.com/paper/2307.10554/) (2023)
- [Mixed-Precision Neural Networks: A Survey](https://mlsanity.com/paper/2208.06064/) (2022)
- [Self-Distilled Quantization: Achieving High Compression Rates in Transformer-Based Language Models](https://mlsanity.com/paper/2307.05972/) (2023)
- [Adaptive quantization with mixed-precision based on low-cost proxy](https://mlsanity.com/paper/2402.17706/) (2024)
- [Width Transfer: On the (In)variance of Width Optimization](https://mlsanity.com/paper/2104.13255/) (2021)
- [CSMPQ:Class Separability Based Mixed-Precision Quantization](https://mlsanity.com/paper/2212.10220/) (2022)

## Abstract (arXiv)

Weight quantization for deep ConvNets has shown promising results for applications such as image classification and semantic segmentation and is especially important for applications where memory storage is limited. However, when aiming for quantization without accuracy degradation, different tasks may end up with different bitwidths. This creates complexity for software and hardware support and the complexity accumulates when one considers mixed-precision quantization, in which case each layer's weights use a different bitwidth. Our key insight is that optimizing for the least bitwidth subject to no accuracy degradation is not necessarily an optimal strategy. This is because one cannot decide optimality between two bitwidths if one has a smaller model size while the other has better accuracy. In this work, we take the first step to understand if some weight bitwidth is better than others by aligning all to the same model size using a width-multiplier. Under this setting, somewhat surprisingly, we show that using a single bitwidth for the whole network can achieve better accuracy compared to mixed-precision quantization targeting zero accuracy degradation when both have the same model size. In particular, our results suggest that when the number of channels becomes a target hyperparameter, a single weight bitwidth throughout the network shows superior results for model compression.