# Improving the Reliability for Confidence Estimation

arXiv: https://arxiv.org/abs/2210.06776 · HTML: https://mlsanity.com/paper/2210.06776/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 52 references that are in the map.
- First posted on arXiv 13 Oct 2022.

## Builds on (most cited first, 30 of 36)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [mixup: Beyond Empirical Risk Minimization](https://mlsanity.com/paper/1710.09412/) (2017)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [The Cityscapes Dataset for Semantic Urban Scene Understanding](https://mlsanity.com/paper/1604.01685/) (2016)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Arbitrary Style Transfer in Real-time with Adaptive Instance Normalization](https://mlsanity.com/paper/1703.06868/) (2017)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [Learning to Generalize: Meta-Learning for Domain Generalization](https://mlsanity.com/paper/1710.03463/) (2017)
- [From Big to Small: Multi-Scale Local Planar Guidance for Monocular Depth Estimation](https://mlsanity.com/paper/1907.10326/) (2019)
- [Calibrating Deep Neural Networks using Focal Loss](https://mlsanity.com/paper/2002.09437/) (2020)
- [Sparsity Invariant CNNs](https://mlsanity.com/paper/1708.06500/) (2017)
- [Gradient Matching for Domain Generalization](https://mlsanity.com/paper/2104.09937/) (2021)
- [Failing Loudly: An Empirical Study of Methods for Detecting Dataset Shift](https://mlsanity.com/paper/1810.11953/) (2018)
- [To Trust Or Not To Trust A Classifier](https://mlsanity.com/paper/1805.11783/) (2018)
- [Addressing Failure Prediction by Learning Model Confidence](https://mlsanity.com/paper/1910.04851/) (2019)
- [Computer Vision for Autonomous Vehicles: Problems, Datasets and State of the Art](https://mlsanity.com/paper/1704.05519/) (2017)
- [Domain Generalization Using a Mixture of Multiple Latent Domains](https://mlsanity.com/paper/1911.07661/) (2019)
- [Confidence-Aware Learning for Deep Neural Networks](https://mlsanity.com/paper/2007.01458/) (2020)
- [Model Adaptation with Synthetic and Real Data for Semantic Dense Foggy Scene Understanding](https://mlsanity.com/paper/1808.01265/) (2018)
- [On Feature Normalization and Data Augmentation](https://mlsanity.com/paper/2002.11102/) (2020)
- [Learning Meta Face Recognition in Unseen Domains](https://mlsanity.com/paper/2003.07733/) (2020)
- [Post-hoc Uncertainty Calibration for Domain Drift Scenarios](https://mlsanity.com/paper/2012.10988/) (2020)
- [Detecting Errors and Estimating Accuracy on Unlabeled Data with Self-training Ensembles](https://mlsanity.com/paper/2106.15728/) (2021)
- [MetaSets: Meta-Learning on Point Sets for Generalizable Representations](https://mlsanity.com/paper/2204.07311/) (2022)
- [Meta Spatio-Temporal Debiasing for Video Scene Graph Generation](https://mlsanity.com/paper/2207.11441/) (2022)

## Built on by (most cited first, 6 of 6)

- [LMC: Large Model Collaboration with Cross-assessment for Training-Free Open-Set Object Recognition](https://mlsanity.com/paper/2309.12780/) (2023)
- [Discretization-Induced Dirichlet Posterior for Robust Uncertainty Quantification on Regression](https://mlsanity.com/paper/2308.09065/) (2023)
- [DECIDER: Leveraging Foundation Model Priors for Improved Model Failure Detection and Explanation](https://mlsanity.com/paper/2408.00331/) (2024)
- [Measuring and Modeling Uncertainty Degree for Monocular Depth Estimation](https://mlsanity.com/paper/2307.09929/) (2023)
- [TRUST: Test-time Resource Utilization for Superior Trustworthiness](https://mlsanity.com/paper/2506.06048/) (2025)
- [Continual Error Correction on Low-Resource Devices](https://mlsanity.com/paper/2511.21652/) (2025)

## Abstract (arXiv)

Confidence estimation, a task that aims to evaluate the trustworthiness of the model's prediction output during deployment, has received lots of research attention recently, due to its importance for the safe deployment of deep models. Previous works have outlined two important qualities that a reliable confidence estimation model should possess, i.e., the ability to perform well under label imbalance and the ability to handle various out-of-distribution data inputs. In this work, we propose a meta-learning framework that can simultaneously improve upon both qualities in a confidence estimation model. Specifically, we first construct virtual training and testing sets with some intentionally designed distribution differences between them. Our framework then uses the constructed sets to train the confidence estimation model through a virtual training and testing scheme leading it to learn knowledge that generalizes to diverse distributions. We show the effectiveness of our framework on both monocular depth estimation and image classification.