# BMPQ: Bit-Gradient Sensitivity Driven Mixed-Precision Quantization of DNNs from Scratch

arXiv: https://arxiv.org/abs/2112.13843 · HTML: https://mlsanity.com/paper/2112.13843/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 26 references that are in the map.
- First posted on arXiv 24 Dec 2021.

## Builds on (most cited first, 20 of 20)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks](https://mlsanity.com/paper/1603.05279/) (2016)
- [Quantizing deep convolutional networks for efficient inference: A whitepaper](https://mlsanity.com/paper/1806.08342/) (2018)
- [PACT: Parameterized Clipping Activation for Quantized Neural Networks](https://mlsanity.com/paper/1805.06085/) (2018)
- [HAQ: Hardware-Aware Automated Quantization with Mixed Precision](https://mlsanity.com/paper/1811.08886/) (2018)
- [HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precision](https://mlsanity.com/paper/1905.03696/) (2019)
- [LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networks](https://mlsanity.com/paper/1807.10029/) (2018)
- [Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights](https://mlsanity.com/paper/1702.03044/) (2017)
- [Sparse Networks from Scratch: Faster Training without Losing Performance](https://mlsanity.com/paper/1907.04840/) (2019)
- [Trained Ternary Quantization](https://mlsanity.com/paper/1612.01064/) (2016)
- [Ternary Weight Networks](https://mlsanity.com/paper/1605.04711/) (2016)
- [Mixed Precision Quantization of ConvNets via Differentiable Neural Architecture Search](https://mlsanity.com/paper/1812.00090/) (2018)
- [Bit-Flip Attack: Crushing Neural Network with Progressive Bit Search](https://mlsanity.com/paper/1903.12269/) (2019)
- [HAWQV3: Dyadic Neural Network Quantization](https://mlsanity.com/paper/2011.10680/) (2020)
- [BSQ: Exploring Bit-Level Sparsity for Mixed-Precision Neural Network Quantization](https://mlsanity.com/paper/2102.10462/) (2021)
- [A Tunable Robust Pruning Framework Through Dynamic Network Rewiring of DNNs](https://mlsanity.com/paper/2011.03083/) (2020)
- [Activation Density based Mixed-Precision Quantization for Energy Efficient Neural Networks](https://mlsanity.com/paper/2101.04354/) (2021)

## Built on by (most cited first, 11 of 11)

- [LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuning](https://mlsanity.com/paper/2311.12023/) (2023)
- [Learning to Linearize Deep Neural Networks for Secure and Efficient Private Inference](https://mlsanity.com/paper/2301.09254/) (2023)
- [CLAMP-ViT: Contrastive Data-Free Learning for Adaptive Post-Training Quantization of ViTs](https://mlsanity.com/paper/2407.05266/) (2024)
- [FIT: A Metric for Model Sensitivity](https://mlsanity.com/paper/2210.08502/) (2022)
- [Self-Attentive Pooling for Efficient Deep Learning](https://mlsanity.com/paper/2209.07659/) (2022)
- [Mixed-Precision Quantization for Deep Vision Models with Integer Quadratic Programming](https://mlsanity.com/paper/2307.05657/) (2023)
- [OuroMamba: A Data-Free Quantization Framework for Vision Mamba](https://mlsanity.com/paper/2503.10959/) (2025)
- [Sparse Mixture Once-for-all Adversarial Training for Efficient In-Situ Trade-Off Between Accuracy and Robustness of DNNs](https://mlsanity.com/paper/2302.03523/) (2022)
- [Block Selective Reprogramming for On-device Training of Vision Transformers](https://mlsanity.com/paper/2405.10951/) (2024)
- [Value-Driven Mixed-Precision Quantization for Patch-Based Inference on Microcontrollers](https://mlsanity.com/paper/2401.13714/) (2024)
- [FAMES: Fast Approximate Multiplier Substitution for Mixed-Precision Quantized DNNs--Down to 2 Bits!](https://mlsanity.com/paper/2411.18055/) (2024)

## Abstract (arXiv)

Large DNNs with mixed-precision quantization can achieve ultra-high compression while retaining high classification performance. However, because of the challenges in finding an accurate metric that can guide the optimization process, these methods either sacrifice significant performance compared to the 32-bit floating-point (FP-32) baseline or rely on a compute-expensive, iterative training policy that requires the availability of a pre-trained baseline. To address this issue, this paper presents BMPQ, a training method that uses bit gradients to analyze layer sensitivities and yield mixed-precision quantized models. BMPQ requires a single training iteration but does not need a pre-trained baseline. It uses an integer linear program (ILP) to dynamically adjust the precision of layers during training, subject to a fixed hardware budget. To evaluate the efficacy of BMPQ, we conduct extensive experiments with VGG16 and ResNet18 on CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets. Compared to the baseline FP-32 models, BMPQ can yield models that have 15.4x fewer parameter bits with a negligible drop in accuracy. Compared to the SOTA "during training", mixed-precision training scheme, our models are 2.1x, 2.2x, and 2.9x smaller, on CIFAR-10, CIFAR-100, and Tiny-ImageNet, respectively, with an improved accuracy of up to 14.54%.