# Top-label calibration and multiclass-to-binary reductions

arXiv: https://arxiv.org/abs/2107.08353 · HTML: https://mlsanity.com/paper/2107.08353/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 45 references that are in the map.
- First posted on arXiv 18 Jul 2021.

## Builds on (most cited first, 21 of 21)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Wide Residual Networks](https://mlsanity.com/paper/1605.07146/) (2016)
- [Inherent Trade-Offs in the Fair Determination of Risk Scores](https://mlsanity.com/paper/1609.05807/) (2016)
- [Measuring Calibration in Deep Learning](https://mlsanity.com/paper/1904.01685/) (2019)
- [Accurate Uncertainties for Deep Learning Using Calibrated Regression](https://mlsanity.com/paper/1807.00263/) (2018)
- [Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibration](https://mlsanity.com/paper/1910.12656/) (2019)
- [Calibrating Deep Neural Networks using Focal Loss](https://mlsanity.com/paper/2002.09437/) (2020)
- [Verified Uncertainty Calibration](https://mlsanity.com/paper/1909.10155/) (2019)
- [Mix-n-Match: Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learning](https://mlsanity.com/paper/2003.07329/) (2020)
- [Evaluating model calibration in classification](https://mlsanity.com/paper/1902.06977/) (2019)
- [Mitigating Bias in Calibration Error Estimation](https://mlsanity.com/paper/2012.08668/) (2020)
- [Distribution-free uncertainty quantification for classification under label shift](https://mlsanity.com/paper/2103.03323/) (2021)
- [Calibration of Neural Networks using Splines](https://mlsanity.com/paper/2006.12800/) (2020)
- [Calibration tests in multi-class classification: A unifying framework](https://mlsanity.com/paper/1910.11385/) (2019)
- [Distribution-free binary classification: prediction sets, confidence intervals and calibration](https://mlsanity.com/paper/2006.10564/) (2020)
- [Intra Order-preserving Functions for Calibration of Multi-Class Neural Networks](https://mlsanity.com/paper/2003.06820/) (2020)
- [Distribution-free calibration guarantees for histogram binning without sample splitting](https://mlsanity.com/paper/2105.04656/) (2021)
- [Multi-Class Uncertainty Calibration via Mutual Information Maximization-based Binning](https://mlsanity.com/paper/2006.13092/) (2020)
- [Concentration Inequalities for Multinoulli Random Variables](https://mlsanity.com/paper/2001.11595/) (2020)

## Built on by (most cited first, 30 of 46)

- [Mitigating Neural Network Overconfidence with Logit Normalization](https://mlsanity.com/paper/2205.09310/) (2022)
- [Calibrated Learning to Defer with One-vs-All Classifiers](https://mlsanity.com/paper/2202.03673/) (2022)
- [Learning to Defer to Multiple Experts: Consistent Surrogate Losses, Confidence Calibration, and Conformal Ensembles](https://mlsanity.com/paper/2210.16955/) (2022)
- [Tracking the risk of a deployed model and detecting harmful distribution shifts](https://mlsanity.com/paper/2110.06177/) (2021)
- [Thermometer: Towards Universal Calibration for Large Language Models](https://mlsanity.com/paper/2403.08819/) (2024)
- [Calibrated Selective Classification](https://mlsanity.com/paper/2208.12084/) (2022)
- [Calibrating Long-form Generations from Large Language Models](https://mlsanity.com/paper/2402.06544/) (2024)
- [Towards Human-AI Complementarity with Prediction Sets](https://mlsanity.com/paper/2405.17544/) (2024)
- [Conformal Semantic Image Segmentation: Post-hoc Quantification of Predictive Uncertainty](https://mlsanity.com/paper/2405.05145/) (2024)
- [On Computationally Efficient Multi-Class Calibration](https://mlsanity.com/paper/2402.07821/) (2024)
- [Confidence Calibration of Classifiers with Many Classes](https://mlsanity.com/paper/2411.02988/) (2024)
- [Minimum-Risk Recalibration of Classifiers](https://mlsanity.com/paper/2305.10886/) (2023)
- [Scaffolding Sets](https://mlsanity.com/paper/2111.03135/) (2021)
- [Online Platt Scaling with Calibeating](https://mlsanity.com/paper/2305.00070/) (2023)
- [A Survey of Calibration Process for Black-Box LLMs](https://mlsanity.com/paper/2412.12767/) (2024)
- [Reducing Overconfidence Predictions for Autonomous Driving Perception](https://mlsanity.com/paper/2202.07825/) (2022)
- [Approaching Test Time Augmentation in the Context of Uncertainty Calibration for Deep Neural Networks](https://mlsanity.com/paper/2304.05104/) (2023)
- [On the Overconfidence Problem in Semantic 3D Mapping](https://mlsanity.com/paper/2311.10018/) (2023)
- [Three Types of Calibration with Properties and their Semantic and Formal Relationships](https://mlsanity.com/paper/2504.18395/) (2025)
- [Calibrating LLMs for Text-to-SQL Parsing by Leveraging Sub-clause Frequencies](https://mlsanity.com/paper/2505.23804/) (2025)
- [Faster online calibration without randomization: interval forecasts and the power of two choices](https://mlsanity.com/paper/2204.13087/) (2022)
- [Class-wise and reduced calibration methods](https://mlsanity.com/paper/2210.03702/) (2022)
- [Stability and Multigroup Fairness in Ranking with Uncertain Predictions](https://mlsanity.com/paper/2402.09326/) (2024)
- [Evaluating multiple models using labeled and unlabeled data](https://mlsanity.com/paper/2501.11866/) (2025)
- [SELFOOD: Self-Supervised Out-Of-Distribution Detection via Learning to Rank](https://mlsanity.com/paper/2305.14696/) (2023)
- [Pearls from Pebbles: Improved Confidence Functions for Auto-labeling](https://mlsanity.com/paper/2404.16188/) (2024)
- [Optimizing Estimators of Squared Calibration Errors in Classification](https://mlsanity.com/paper/2410.07014/) (2024)
- [Random Forest Calibration](https://mlsanity.com/paper/2501.16756/) (2025)
- [Robust Decision Making with Partially Calibrated Forecasts](https://mlsanity.com/paper/2510.23471/) (2025)
- [Improving Multi-Class Calibration through Normalization-Aware Isotonic Techniques](https://mlsanity.com/paper/2512.09054/) (2025)

## Abstract (arXiv)

A multiclass classifier is said to be top-label calibrated if the reported probability for the predicted class -- the top-label -- is calibrated, conditioned on the top-label. This conditioning on the top-label is absent in the closely related and popular notion of confidence calibration, which we argue makes confidence calibration difficult to interpret for decision-making. We propose top-label calibration as a rectification of confidence calibration. Further, we outline a multiclass-to-binary (M2B) reduction framework that unifies confidence, top-label, and class-wise calibration, among others. As its name suggests, M2B works by reducing multiclass calibration to numerous binary calibration problems, each of which can be solved using simple binary calibration routines. We instantiate the M2B framework with the well-studied histogram binning (HB) binary calibrator, and prove that the overall procedure is multiclass calibrated without making any assumptions on the underlying data distribution. In an empirical evaluation with four deep net architectures on CIFAR-10 and CIFAR-100, we find that the M2B + HB procedure achieves lower top-label and class-wise calibration error than other approaches such as temperature scaling. Code for this work is available at urlhttps://github.com/aigen/df-posthoc-calibration.