# Non-Parametric Calibration for Classification

arXiv: https://arxiv.org/abs/1906.04933 · HTML: https://mlsanity.com/paper/1906.04933/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 Jun 2019.

## Built on by (most cited first, 30 of 57)

- [A Survey of Uncertainty in Deep Neural Networks](https://mlsanity.com/paper/2107.03342/) (2021)
- [Soft Calibration Objectives for Neural Networks](https://mlsanity.com/paper/2108.00106/) (2021)
- [Classifier Calibration: A survey on how to assess and improve predicted class probabilities](https://mlsanity.com/paper/2112.10327/) (2021)
- [Better Uncertainty Calibration via Proper Scores for Classification and Beyond](https://mlsanity.com/paper/2203.07835/) (2022)
- [Parameterized Temperature Scaling for Boosting the Expressive Power in Post-Hoc Uncertainty Calibration](https://mlsanity.com/paper/2102.12182/) (2021)
- [A Consistent and Differentiable Lp Canonical Calibration Error Estimator](https://mlsanity.com/paper/2210.07810/) (2022)
- [LitCab: Lightweight Language Model Calibration over Short- and Long-form Responses](https://mlsanity.com/paper/2310.19208/) (2023)
- [Meta-Cal: Well-controlled Post-hoc Calibration by Ranking](https://mlsanity.com/paper/2105.04290/) (2021)
- [Spatially Varying Label Smoothing: Capturing Uncertainty from Expert Annotations](https://mlsanity.com/paper/2104.05788/) (2021)
- [Robust Models are less Over-Confident](https://mlsanity.com/paper/2210.05938/) (2022)
- [Adaptive Temperature Scaling for Robust Calibration of Deep Neural Networks](https://mlsanity.com/paper/2208.00461/) (2022)
- [Beyond In-Domain Scenarios: Robust Density-Aware Calibration](https://mlsanity.com/paper/2302.05118/) (2023)
- [Two Sides of Miscalibration: Identifying Over and Under-Confidence Prediction for Network Calibration](https://mlsanity.com/paper/2308.03172/) (2023)
- [Trust Your Robots! Predictive Uncertainty Estimation of Neural Networks with Sparse Gaussian Processes](https://mlsanity.com/paper/2109.09690/) (2021)
- [Transfer Knowledge from Head to Tail: Uncertainty Calibration under Long-tailed Distribution](https://mlsanity.com/paper/2304.06537/) (2023)
- [On the relationship between calibrated predictors and unbiased volume estimation](https://mlsanity.com/paper/2112.12560/) (2021)
- [Consistent and Asymptotically Unbiased Estimation of Proper Calibration Errors](https://mlsanity.com/paper/2312.08589/) (2023)
- [SCRIB: Set-classifier with Class-specific Risk Bounds for Blackbox Models](https://mlsanity.com/paper/2103.03945/) (2021)
- [Minimum-Risk Recalibration of Classifiers](https://mlsanity.com/paper/2305.10886/) (2023)
- [Beyond Probability Partitions: Calibrating Neural Networks with Semantic Aware Grouping](https://mlsanity.com/paper/2306.04985/) (2023)
- [Taking a Step Back with KCal: Multi-Class Kernel-Based Calibration for Deep Neural Networks](https://mlsanity.com/paper/2202.07679/) (2022)
- [Calibrating a Deep Neural Network with Its Predecessors](https://mlsanity.com/paper/2302.06245/) (2023)
- [On Calibrated Model Uncertainty in Deep Learning](https://mlsanity.com/paper/2206.07795/) (2022)
- [GraphCleaner: Detecting Mislabelled Samples in Popular Graph Learning Benchmarks](https://mlsanity.com/paper/2306.00015/) (2023)
- [Animal Identification with Independent Foreground and Background Modeling](https://mlsanity.com/paper/2408.12930/) (2024)
- [Active Assessment of Prediction Services as Accuracy Surface Over Attribute Combinations](https://mlsanity.com/paper/2108.06514/) (2021)
- [Towards Certification of Uncertainty Calibration under Adversarial Attacks](https://mlsanity.com/paper/2405.13922/) (2024)
- [Confidence-Aware Multi-Field Model Calibration](https://mlsanity.com/paper/2402.17655/) (2024)
- [PAC-Bayes Analysis for Recalibration in Classification](https://mlsanity.com/paper/2406.06227/) (2024)
- [Uncertainty Calibration with Energy Based Instance-wise Scaling in the Wild Dataset](https://mlsanity.com/paper/2407.12330/) (2024)

## Abstract (arXiv)

Many applications of classification methods not only require high accuracy but also reliable estimation of predictive uncertainty. However, while many current classification frameworks, in particular deep neural networks, achieve high accuracy, they tend to incorrectly estimate uncertainty. In this paper, we propose a method that adjusts the confidence estimates of a general classifier such that they approach the probability of classifying correctly. In contrast to existing approaches, our calibration method employs a non-parametric representation using a latent Gaussian process, and is specifically designed for multi-class classification. It can be applied to any classifier that outputs confidence estimates and is not limited to neural networks. We also provide a theoretical analysis regarding the over- and underconfidence of a classifier and its relationship to calibration, as well as an empirical outlook for calibrated active learning. In experiments we show the universally strong performance of our method across different classifiers and benchmark data sets, in particular for state-of-the art neural network architectures.