# Multi-Class Uncertainty Calibration via Mutual Information Maximization-based Binning

arXiv: https://arxiv.org/abs/2006.13092 · HTML: https://mlsanity.com/paper/2006.13092/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jun 2020.

## Built on by (most cited first, 27 of 27)

- [Classifier Calibration: A survey on how to assess and improve predicted class probabilities](https://mlsanity.com/paper/2112.10327/) (2021)
- [Top-label calibration and multiclass-to-binary reductions](https://mlsanity.com/paper/2107.08353/) (2021)
- [Rethinking Confidence Calibration for Failure Prediction](https://mlsanity.com/paper/2303.02970/) (2023)
- [Combining Human Predictions with Model Probabilities via Confusion Matrices and Calibration](https://mlsanity.com/paper/2109.14591/) (2021)
- [ConfTuner: Training Large Language Models to Express Their Confidence Verbally](https://mlsanity.com/paper/2508.18847/) (2025)
- [Meta-Cal: Well-controlled Post-hoc Calibration by Ranking](https://mlsanity.com/paper/2105.04290/) (2021)
- [On Calibrating Semantic Segmentation Models: Analyses and An Algorithm](https://mlsanity.com/paper/2212.12053/) (2022)
- [Proximity-Informed Calibration for Deep Neural Networks](https://mlsanity.com/paper/2306.04590/) (2023)
- [FairCal: Fairness Calibration for Face Verification](https://mlsanity.com/paper/2106.03761/) (2021)
- [A Benchmark Study on Calibration](https://mlsanity.com/paper/2308.11838/) (2023)
- [Confidence Calibration of Classifiers with Many Classes](https://mlsanity.com/paper/2411.02988/) (2024)
- [Revisiting Confidence Estimation: Towards Reliable Failure Prediction](https://mlsanity.com/paper/2403.02886/) (2024)
- [Taking a Step Back with KCal: Multi-Class Kernel-Based Calibration for Deep Neural Networks](https://mlsanity.com/paper/2202.07679/) (2022)
- [Improving Uncertainty of Deep Learning-based Object Classification on Radar Spectra using Label Smoothing](https://mlsanity.com/paper/2109.12851/) (2021)
- [Optimizing Estimators of Squared Calibration Errors in Classification](https://mlsanity.com/paper/2410.07014/) (2024)
- [Enhancing Out-of-Distribution Detection with Extended Logit Normalization](https://mlsanity.com/paper/2504.11434/) (2025)
- [Improving Multi-Class Calibration through Normalization-Aware Isotonic Techniques](https://mlsanity.com/paper/2512.09054/) (2025)
- [Investigation of Uncertainty of Deep Learning-based Object Classification on Radar Spectra](https://mlsanity.com/paper/2106.05870/) (2021)
- [Uncertainty Calibration for Deep Audio Classifiers](https://mlsanity.com/paper/2206.13071/) (2022)
- [Layer-Stack Temperature Scaling](https://mlsanity.com/paper/2211.10193/) (2022)
- [Making Heads and Tails of Models with Marginal Calibration for Sparse Tagsets](https://mlsanity.com/paper/2109.07494/) (2021)
- [Hidden Heterogeneity: When to Choose Similarity-Based Calibration](https://mlsanity.com/paper/2202.01840/) (2022)
- [Perception and Semantic Aware Regularization for Sequential Confidence Calibration](https://mlsanity.com/paper/2305.19498/) (2023)
- [Tilt and Average : Geometric Adjustment of the Last Layer for Recalibration](https://mlsanity.com/paper/2406.10017/) (2024)
- [Combining Priors with Experience: Confidence Calibration Based on Binomial Process Modeling](https://mlsanity.com/paper/2412.10658/) (2024)
- [Scalable Utility-Aware Multiclass Calibration](https://mlsanity.com/paper/2510.25458/) (2025)
- [Confidence Calibration with Bounded Error Using Transformations](https://mlsanity.com/paper/2102.12680/) (2021)

## Abstract (arXiv)

Post-hoc multi-class calibration is a common approach for providing high-quality confidence estimates of deep neural network predictions. Recent work has shown that widely used scaling methods underestimate their calibration error, while alternative Histogram Binning (HB) methods often fail to preserve classification accuracy. When classes have small prior probabilities, HB also faces the issue of severe sample-inefficiency after the conversion into K one-vs-rest class-wise calibration problems. The goal of this paper is to resolve the identified issues of HB in order to provide calibrated confidence estimates using only a small holdout calibration dataset for bin optimization while preserving multi-class ranking accuracy. From an information-theoretic perspective, we derive the I-Max concept for binning, which maximizes the mutual information between labels and quantized logits. This concept mitigates potential loss in ranking performance due to lossy quantization, and by disentangling the optimization of bin edges and representatives allows simultaneous improvement of ranking and calibration performance. To improve the sample efficiency and estimates from a small calibration set, we propose a shared class-wise (sCW) calibration strategy, sharing one calibrator among similar classes (e.g., with similar class priors) so that the training sets of their class-wise calibration problems can be merged to train the single calibrator. The combination of sCW and I-Max binning outperforms the state of the art calibration methods on various evaluation metrics across different benchmark datasets and models, using a small calibration set (e.g., 1k samples for ImageNet).