# On Calibration of Modern Quantized Efficient Neural Networks

arXiv: https://arxiv.org/abs/2309.13866 · HTML: https://mlsanity.com/paper/2309.13866/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 29 references that are in the map.
- First posted on arXiv 25 Sep 2023.

## Builds on (most cited first, 21 of 21)

- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classification](https://mlsanity.com/paper/1502.01852/) (2015)
- [Searching for MobileNetV3](https://mlsanity.com/paper/1905.02244/) (2019)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [MedMNIST v2 -- A large-scale lightweight benchmark for 2D and 3D biomedical image classification](https://mlsanity.com/paper/2110.14795/) (2021)
- [A White Paper on Neural Network Quantization](https://mlsanity.com/paper/2106.08295/) (2021)
- [GhostNet: More Features from Cheap Operations](https://mlsanity.com/paper/1911.11907/) (2019)
- [HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precision](https://mlsanity.com/paper/1905.03696/) (2019)
- [Revisiting the Calibration of Modern Neural Networks](https://mlsanity.com/paper/2106.07998/) (2021)
- [Incremental Network Quantization: Towards Lossless CNNs with Low-Precision Weights](https://mlsanity.com/paper/1702.03044/) (2017)
- [Differentiable Model Compression via Pseudo Quantization Noise](https://mlsanity.com/paper/2104.09987/) (2021)
- [HMQ: Hardware Friendly Mixed Precision Quantization Block for CNNs](https://mlsanity.com/paper/2007.09952/) (2020)
- [Well-calibrated Model Uncertainty with Temperature Scaling for Dropout Variational Inference](https://mlsanity.com/paper/1909.13550/) (2019)
- [Beyond calibration: estimating the grouping loss of modern neural networks](https://mlsanity.com/paper/2210.16315/) (2022)
- [Sharpness-aware Quantization for Deep Neural Networks](https://mlsanity.com/paper/2111.12273/) (2021)
- [Do All MobileNets Quantize Poorly? Gaining Insights into the Effect of Quantization on Depthwise Separable Convolutional Networks Through the Eyes of Multi-scale Distributional Dynamics](https://mlsanity.com/paper/2104.11849/) (2021)
- [Towards unraveling calibration biases in medical image analysis](https://mlsanity.com/paper/2305.05101/) (2023)
- [An Underexplored Dilemma between Confidence and Calibration in Quantized Neural Networks](https://mlsanity.com/paper/2111.08163/) (2021)
- [AugShuffleNet: Communicate More, Compute Less](https://mlsanity.com/paper/2203.06589/) (2022)

## Built on by (most cited first, 1 of 1)

- [Hierarchical Selective Classification](https://mlsanity.com/paper/2405.11533/) (2024)

## Abstract (arXiv)

We explore calibration properties at various precisions for three architectures: ShuffleNetv2, GhostNet-VGG, and MobileOne; and two datasets: CIFAR-100 and PathMNIST. The quality of calibration is observed to track the quantization quality; it is well-documented that performance worsens with lower precision, and we observe a similar correlation with poorer calibration. This becomes especially egregious at 4-bit activation regime. GhostNet-VGG is shown to be the most robust to overall performance drop at lower precision. We find that temperature scaling can improve calibration error for quantized networks, with some caveats. We hope that these preliminary insights can lead to more opportunities for explainable and reliable EdgeML.