mlsanityData to 7 Oct 2026

What classifiers know what they don't?

arXiv 2107.06217 ↗First posted 13 Jul 2021cs.LG cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

Being uncertain when facing the unknown is key to intelligent decision making. However, machine learning algorithms lack reliable estimates about their predictive uncertainty. This leads to wrong and overly-confident decisions when encountering classes unseen during training. Despite the importance of equipping classifiers with uncertainty estimates ready for the real world, prior work has focused on small datasets and little or no class discrepancy between training and testing data. To close this gap, we introduce UIMNET: a realistic, ImageNet-scale test-bed to evaluate predictive uncertainty estimates for deep image classifiers. Our benchmark provides implementations of eight state-of-the-art algorithms, six uncertainty measures, four in-domain metrics, three out-domain metrics, and a fully automated pipeline to train, calibrate, ensemble, select, and evaluate models. Our test-bed is open-source and all of our results are reproducible from a fixed commit in our repository. Adding new datasets, algorithms, measures, or metrics is a matter of a few lines of code-in so hoping that UIMNET becomes a stepping stone towards realistic, rigorous, and reproducible research in uncertainty estimation. Our results show that ensembles of ERM classifiers as well as single MIMO classifiers are the two best alternatives currently available to measure uncertainty about both in-domain and out-domain classes.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 30 of its 53 references that are in the map. First posted on arXiv 13 Jul 2021.

Builds on

30 of 53 references.

References · 30

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  6. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  7. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  8. 2017On Calibration of Modern Neural Networkscited by 4,860
  9. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  10. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  11. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  12. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
18 more
  1. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  2. 2019Natural Adversarial Examplescited by 1,401
  3. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  4. 2020In Search of Lost Domain Generalizationcited by 1,042
  5. 2018Exploration by Random Network Distillationcited by 966
  6. 2018GPyTorch: Blackbox Matrix-Matrix Gaussian Process Inference with GPU Accelerationcited by 603
  7. 2020A Unifying Review of Deep and Shallow Anomaly Detectioncited by 423
  8. 2020Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awarenesscited by 389
  9. 2019Deep Learning for Anomaly Detection: A Surveycited by 357
  10. 2018Why ReLU networks yield high-confidence predictions far away from the training data and how to mitigate the problemcited by 323
  11. 2019On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networkscited by 304
  12. 2020Pitfalls of In-Domain Uncertainty Estimation and Ensembling in Deep Learningcited by 208
  13. 2018Sensitivity and Generalization in Neural Networks: an Empirical Studycited by 187
  14. 2020Training independent subnetworks for robust predictioncited by 177
  15. 2020BREEDS: Benchmarks for Subpopulation Shiftcited by 155
  16. 2017A causal framework for explaining the predictions of black-box sequence-to-sequence modelscited by 58
  17. 2018Evaluating Uncertainty Quantification in End-to-End Autonomous Driving Controlcited by 54
  18. 2019Model-Predictive Policy Learning with Uncertainty Regularization for Driving in Dense Trafficcited by 50

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 2; 2024: 1; 2025: 1; 2026: 1.

6 papers.

  1. 2023How to Fix a Broken Confidence Estimator: Evaluating Post-hoc Methods for Selective Classification with Deep Neural Networkscited by 15
  2. 2024A Novel Characterization of the Population Area Under the Risk Coverage Curve (AURC) and Rates of Finite Sample Estimatorscited by 13
  3. 2022Understanding out-of-distribution accuracies through quantifying difficulty of test samplescited by 3
  4. 2023Unified Uncertainty Calibrationcited by 0
  5. 2025Revisiting Reweighted Risk for Calibration: AURC, Focal, and Inverse Focal Losscited by 0
  6. 2026Post-hoc Selective Classification for Reliable Synthetic Image Detectioncited by 0