# Confidence Calibration of Deep Learning Systems

arXiv: https://arxiv.org/abs/2608.12100 · HTML: https://mlsanity.com/paper/2608.12100/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 14 of its 121 references that are in the map.
- First posted on arXiv 12 Aug 2026.

## Builds on (most cited first, 14 of 14)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [When Does Label Smoothing Help?](https://mlsanity.com/paper/1906.02629/) (2019)
- [VisDA: The Visual Domain Adaptation Challenge](https://mlsanity.com/paper/1710.06924/) (2017)
- [Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations](https://mlsanity.com/paper/2110.12088/) (2021)
- [Posterior calibration and exploratory analysis for natural language processing models](https://mlsanity.com/paper/1508.05154/) (2015)
- [Robust Long-Tailed Learning under Label Noise](https://mlsanity.com/paper/2108.11569/) (2021)
- [Adaptive conformal classification with noisy labels](https://mlsanity.com/paper/2309.05092/) (2023)
- [Label Noise Robustness of Conformal Prediction](https://mlsanity.com/paper/2209.14295/) (2022)
- [Private Prediction Sets](https://mlsanity.com/paper/2102.06202/) (2021)
- [Locally Private Mean Estimation: Z-test and Tight Confidence Intervals](https://mlsanity.com/paper/1810.08054/) (2018)
- [Unsupervised Calibration under Covariate Shift](https://mlsanity.com/paper/2006.16405/) (2020)
- [Differential Privacy in the Shuffle Model: A Survey of Separations](https://mlsanity.com/paper/2107.11839/) (2021)
- [Conformal Prediction of Classifiers with Many Classes based on Noisy Labels](https://mlsanity.com/paper/2501.12749/) (2025)
- [AnomMAN: Detect Anomaly on Multi-view Attributed Networks](https://mlsanity.com/paper/2201.02822/) (2022)

## Abstract (arXiv)

In high-stakes applications, reliable confidence estimates are as important as the predictions themselves. Confidence calibration ensures that predicted probabilities reflect the likelihood of correctness, making it essential for safe deployment of deep learning models. However, existing methods typically assume access to clean validation data, which is often unrealistic due to label noise and domain shifts. This thesis develops methods for improving calibration under these conditions. First, we address calibration under label noise. Standard methods can produce misleading confidence estimates when labels are unreliable. We propose a framework that uses an estimated noise model to reconstruct noise-free confidence estimates by modeling the relationship between noisy and clean label distributions. We extend this approach to Conformal Prediction (CP), which provides set-valued predictions with guaranteed coverage. Our noise-aware CP method estimates clean conformity scores despite label noise, enabling reliable uncertainty quantification. Next, we study calibration in unsupervised domain adaptation, where a model trained on a labeled source domain is adapted to an unlabeled target domain. Since labeled target data are unavailable, we estimate target-domain accuracy from source performance and domain discrepancies, enabling calibration without target labels. We also consider privacy-preserving settings in which user labels and model outputs must remain protected. We propose a locally differentially private conformal prediction framework that provides valid uncertainty quantification while maintaining privacy guarantees and balancing privacy, computational feasibility, and prediction reliability. Our results bridge calibration theory and practical deployment in safety-critical applications, contributing to reliable, privacy-preserving, and noise-resilient neural network predictions.