mlsanityData to 7 Oct 2026

Unified Uncertainty Calibration

arXiv 2310.01202 ↗First posted 2 Oct 2023stat.ML cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

To build robust, fair, and safe AI systems, we would like our classifiers to say ``I don't know'' when facing test examples that are difficult or fall outside of the training classes.The ubiquitous strategy to predict under uncertainty is the simplistic reject-or-classify rule: abstain from prediction if epistemic uncertainty is high, classify otherwise.Unfortunately, this recipe does not allow different sources of uncertainty to communicate with each other, produces miscalibrated predictions, and it does not allow to correct for misspecifications in our uncertainty estimates. To address these three issues, we introduce unified uncertainty calibration (U2C), a holistic framework to combine aleatoric and epistemic uncertainties. U2C enables a clean learning-theoretical analysis of uncertainty estimation, and outperforms reject-or-classify across a variety of ImageNet benchmarks. Our code is available at: https://github.com/facebookresearch/UnifiedUncertaintyCalibration

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 41 of its 53 references that are in the map. First posted on arXiv 2 Oct 2023.

Builds on

41 of 53 references.

References · 41

  1. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  2. 2017On Calibration of Modern Neural Networkscited by 4,860
  3. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  4. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  5. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  6. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  7. 2013Describing Textures in the Wildcited by 2,110
  8. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  9. 2019Invariant Risk Minimizationcited by 1,832
  10. 2016Concrete Problems in AI Safetycited by 1,582
  11. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  12. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,472
29 more
  1. 2019Natural Adversarial Examplescited by 1,401
  2. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  3. 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 903
  4. 2020A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challengescited by 813
  5. 2021Generalized Out-of-Distribution Detection: A Surveycited by 796
  6. 2021A Survey of Uncertainty in Deep Neural Networkscited by 675
  7. 2022Out-of-Distribution Detection with Deep Nearest Neighborscited by 596
  8. 2019Conformal Prediction Under Covariate Shiftcited by 511
  9. 2019Using Self-Supervised Learning Can Improve Model Robustness and Uncertaintycited by 511
  10. 2019Scaling Out-of-Distribution Detection for Real-World Settingscited by 465
  11. 2020A Unifying Review of Deep and Shallow Anomaly Detectioncited by 423
  12. 2022ViM: Out-Of-Distribution with Virtual-logit Matchingcited by 404
  13. 2021Open-Set Recognition: a Good Closed-Set Classifier is All You Need?cited by 351
  14. 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 349
  15. 2018Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problemcited by 323
  16. 2022OpenOOD: Benchmarking Generalized Out-of-Distribution Detectioncited by 267
  17. 2020Gradient Starvation: A Learning Proclivity in Neural Networkscited by 262
  18. 2022Mitigating Neural Network Overconfidence with Logit Normalizationcited by 251
  19. 2021MOS: Towards Scaling Out-of-distribution Detection for Large Semantic Spacecited by 233
  20. 2022Extremely Simple Activation Shaping for Out-of-Distribution Detectioncited by 212
  21. 2021A Simple Fix to Mahalanobis Distance for Improving Near-OOD Detectioncited by 206
  22. 2023OpenOOD v1.5: Enhanced Benchmark for Out-of-Distribution Detectioncited by 178
  23. 2020Training independent subnetworks for robust predictioncited by 177
  24. 2021On Calibration and Out-of-domain Generalizationcited by 134
  25. 2021Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learningcited by 81
  26. 2022Robust Calibration with Multi-domain Temperature Scalingcited by 39
  27. 2020Know Your Limits: Uncertainty Estimation with ReLU Classifiers Fails at Reliable OOD Detectioncited by 26
  28. 2022Estimating Soft Labels for Out-of-Domain Intent Detectioncited by 11
  29. 2021What classifiers know what they don't?cited by 6

Built on by

Not cited yet.