# Calibrating Deep Neural Network Classifiers on Out-of-Distribution Datasets

arXiv: https://arxiv.org/abs/2006.08914 · HTML: https://mlsanity.com/paper/2006.08914/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Jun 2020.

## Built on by (most cited first, 7 of 7)

- [Soft Calibration Objectives for Neural Networks](https://mlsanity.com/paper/2108.00106/) (2021)
- [Confidence-based Estimators for Predictive Performance in Model Monitoring](https://mlsanity.com/paper/2407.08649/) (2024)
- [Responsible and Regulatory Conform Machine Learning for Medicine: A Survey of Challenges and Solutions](https://mlsanity.com/paper/2107.09546/) (2021)
- [O2D2: Out-Of-Distribution Detector to Capture Undecidable Trials in Authorship Verification](https://mlsanity.com/paper/2106.15825/) (2021)
- [Gradient-free variational learning with conditional mixture networks](https://mlsanity.com/paper/2408.16429/) (2024)
- [Frustratingly Easy Uncertainty Estimation for Distribution Shift](https://mlsanity.com/paper/2106.03762/) (2021)
- [Graceful Degradation and Related Fields](https://mlsanity.com/paper/2106.11119/) (2021)

## Abstract (arXiv)

To increase the trustworthiness of deep neural network (DNN) classifiers, an accurate prediction confidence that represents the true likelihood of correctness is crucial. Towards this end, many post-hoc calibration methods have been proposed to leverage a lightweight model to map the target DNN's output layer into a calibrated confidence. Nonetheless, on an out-of-distribution (OOD) dataset in practice, the target DNN can often mis-classify samples with a high confidence, creating significant challenges for the existing calibration methods to produce an accurate confidence. In this paper, we propose a new post-hoc confidence calibration method, called CCAC (Confidence Calibration with an Auxiliary Class), for DNN classifiers on OOD datasets. The key novelty of CCAC is an auxiliary class in the calibration model which separates mis-classified samples from correctly classified ones, thus effectively mitigating the target DNN's being confidently wrong. We also propose a simplified version of CCAC to reduce free parameters and facilitate transfer to a new unseen dataset. Our experiments on different DNN models, datasets and applications show that CCAC can consistently outperform the prior post-hoc calibration methods.