# Mixed Precision DNNs: All you need is a good parametrization

arXiv: https://arxiv.org/abs/1905.11452 · HTML: https://mlsanity.com/paper/1905.11452/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 May 2019.

## Built on by (most cited first, 18 of 18)

- [A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classification](https://mlsanity.com/paper/2205.07877/) (2022)
- [Optimizing Large Language Model Training Using FP4 Quantization](https://mlsanity.com/paper/2501.17116/) (2025)
- [Inspect, Understand, Overcome: A Survey of Practical Methods for AI Safety](https://mlsanity.com/paper/2104.14235/) (2021)
- [Bit-Mixer: Mixed-precision networks with runtime bit-width selection](https://mlsanity.com/paper/2103.17267/) (2021)
- [Quantization and Deployment of Deep Neural Networks on Microcontrollers](https://mlsanity.com/paper/2105.13331/) (2021)
- [Automatic Joint Structured Pruning and Quantization for Efficient Neural Network Training and Compression](https://mlsanity.com/paper/2502.16638/) (2025)
- [Generalizable Mixed-Precision Quantization via Attribution Rank Preservation](https://mlsanity.com/paper/2108.02720/) (2021)
- [Mixed-Precision Neural Networks: A Survey](https://mlsanity.com/paper/2208.06064/) (2022)
- [Resource-efficient Deep Neural Networks for Automotive Radar Interference Mitigation](https://mlsanity.com/paper/2201.10360/) (2022)
- [FLIQS: One-Shot Mixed-Precision Floating-Point and Integer Quantization Search](https://mlsanity.com/paper/2308.03290/) (2023)
- [Quantized Sparse Weight Decomposition for Neural Network Compression](https://mlsanity.com/paper/2207.11048/) (2022)
- [Training Multi-bit Quantized and Binarized Networks with A Learnable Symmetric Quantizer](https://mlsanity.com/paper/2104.00210/) (2021)
- [End-to-end Keyword Spotting using Neural Architecture Search and Quantization](https://mlsanity.com/paper/2104.06666/) (2021)
- [Towards Accurate and Efficient Sub-8-Bit Integer Training](https://mlsanity.com/paper/2411.10948/) (2024)
- [DNN Quantization with Attention](https://mlsanity.com/paper/2103.13322/) (2021)
- [Learning Accurate Performance Predictors for Ultrafast Automated Model Compression](https://mlsanity.com/paper/2304.06393/) (2023)
- [SQuaT: Self-Supervised Knowledge Distillation via Student-Aware Quantized Teacher Features](https://mlsanity.com/paper/2608.10709/) (2026)
- [StatQAT: Statistical Quantizer Optimization for Deep Networks](https://mlsanity.com/paper/2605.17745/) (2026)

## Abstract (arXiv)

Efficient deep neural network (DNN) inference on mobile or embedded devices typically involves quantization of the network parameters and activations. In particular, mixed precision networks achieve better performance than networks with homogeneous bitwidth for the same size constraint. Since choosing the optimal bitwidths is not straight forward, training methods, which can learn them, are desirable. Differentiable quantization with straight-through gradients allows to learn the quantizer's parameters using gradient methods. We show that a suited parametrization of the quantizer is the key to achieve a stable training and a good final performance. Specifically, we propose to parametrize the quantizer with the step size and dynamic range. The bitwidth can then be inferred from them. Other parametrizations, which explicitly use the bitwidth, consistently perform worse. We confirm our findings with experiments on CIFAR-10 and ImageNet and we obtain mixed precision DNNs with learned quantization parameters, achieving state-of-the-art performance.