mlsanityData to 7 Oct 2026

Revisiting Confidence Estimation: Towards Reliable Failure Prediction

arXiv 2403.02886 ↗First posted 5 Mar 2024cs.CV cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Reliable confidence estimation is a challenging yet fundamental requirement in many risk-sensitive applications. However, modern deep neural networks are often overconfident for their incorrect predictions, i.e., misclassified samples from known classes, and out-of-distribution (OOD) samples from unknown classes. In recent years, many confidence calibration and OOD detection methods have been developed. In this paper, we find a general, widely existing but actually-neglected phenomenon that most confidence estimation methods are harmful for detecting misclassification errors. We investigate this problem and reveal that popular calibration and OOD detection methods often lead to worse confidence separation between correctly classified and misclassified examples, making it difficult to decide whether to trust a prediction or not. Finally, we propose to enlarge the confidence gap by finding flat minima, which yields state-of-the-art failure prediction performance under various settings including balanced, long-tailed, and covariate-shift classification scenarios. Our study not only provides a strong baseline for reliable confidence estimation but also acts as a bridge between understanding calibration, OOD detection, and failure prediction. The code is available at urlhttps://github.com/Impression2805/FMFP.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 86 of its 128 references that are in the map. First posted on arXiv 5 Mar 2024.

Builds on

86 of 128 references.

References · 86

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2016Densely Connected Convolutional Networkscited by 6,691
  5. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  6. 2017On Calibration of Modern Neural Networkscited by 4,860
  7. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  8. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  9. 2016Wide Residual Networkscited by 3,057
  10. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  11. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  12. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
38 more
  1. 2013Describing Textures in the Wildcited by 2,110
  2. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  3. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,472
  4. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  5. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  6. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  7. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,350
  8. 2017Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networkscited by 1,325
  9. 2020Energy-based Out-of-distribution Detectioncited by 1,296
  10. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  11. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  12. 2016End to End Learning for Self-Driving Carscited by 1,000
  13. 2019Learning Imbalanced Datasets with Label-Distribution-Aware Margin Losscited by 987
  14. 2019When Does Label Smoothing Help?cited by 985
  15. 2018Deep Anomaly Detection with Outlier Exposurecited by 939
  16. 2019AugMix: A Simple Data Processing Method to Improve Robustness and Uncertaintycited by 897
  17. 2017Selective Classification for Deep Neural Networkscited by 799
  18. 2021Generalized Out-of-Distribution Detection: A Surveycited by 796
  19. 2020Long-tail learning via logit adjustmentcited by 605
  20. 2017Exploring Generalization in Deep Learningcited by 595
  21. 2022Continual Test-Time Domain Adaptationcited by 520
  22. 2020Overfitting in adversarially robust deep learningcited by 504
  23. 2021ReAct: Out-of-distribution Detection With Rectified Activationscited by 486
  24. 2019Scaling Out-of-Distribution Detection for Real-World Settingscited by 465
  25. 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 461
  26. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 431
  27. 2021Revisiting the Calibration of Modern Neural Networkscited by 387
  28. 2019Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibrationcited by 359
  29. 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
  30. 2020Calibrating Deep Neural Networks using Focal Losscited by 353
  31. 2021On the Importance of Gradients for Detecting Distributional Shifts in the Wildcited by 338
  32. 2019On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networkscited by 304
  33. 2022Mitigating Neural Network Overconfidence with Logit Normalizationcited by 251
  34. 2017Emergence of Invariance and Disentanglement in Deep Representationscited by 213
  35. 2021Improving Calibration for Long-Tailed Recognitioncited by 212
  36. 2018To Trust Or Not To Trust A Classifiercited by 205
  37. 2019Addressing Failure Prediction by Learning Model Confidencecited by 183
  38. 2017Computer Vision for Autonomous Vehicles: Problems, Datasets and State of the Artcited by 182

50 most cited of 86 All 86 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 2; 2026: 3.

9 papers.

  1. 2024Overcoming Common Flaws in the Evaluation of Selective Classification Systemscited by 31
  2. 2024Local-Prompt: Extensible Local Prompts for Few-Shot Out-of-Distribution Detectioncited by 14
  3. 2024A Novel Characterization of the Population Area Under the Risk Coverage Curve (AURC) and Rates of Finite Sample Estimatorscited by 13
  4. 2024Towards Understanding Why Label Smoothing Degrades Selective Classification and How to Fix Itcited by 11
  5. 2025Towards Efficient and General-Purpose Few-Shot Misclassification Detection for Vision-Language Modelscited by 1
  6. 2025TrustLoRA: Low-Rank Adaptation for Failure Detection under Out-of-distribution Datacited by 0
  7. 2026From Misclassifications to Outliers: Joint Reliability Assessment in Classificationcited by 0
  8. 2026Expectation Consistency Loss: Rethink Confidence Calibration under Covariate Shiftcited by 0
  9. 2026Pool-Select-Refine for Allocation-Aware Generative Dataset Distillationcited by 0