# A Call to Reflect on Evaluation Practices for Failure Detection in Image Classification

arXiv: https://arxiv.org/abs/2211.15259 · HTML: https://mlsanity.com/paper/2211.15259/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 59 references that are in the map.
- First posted on arXiv 28 Nov 2022.

## Builds on (most cited first, 30 of 40)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [SGDR: Stochastic Gradient Descent with Warm Restarts](https://mlsanity.com/paper/1608.03983/) (2016)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Simple and Scalable Predictive Uncertainty Estimation using Deep Ensembles](https://mlsanity.com/paper/1612.01474/) (2016)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Shortcut Learning in Deep Neural Networks](https://mlsanity.com/paper/2004.07780/) (2020)
- [A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks](https://mlsanity.com/paper/1807.03888/) (2018)
- [WILDS: A Benchmark of in-the-Wild Distribution Shifts](https://mlsanity.com/paper/2012.07421/) (2020)
- [Enhancing The Reliability of Out-of-distribution Image Detection in Neural Networks](https://mlsanity.com/paper/1706.02690/) (2017)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [Selective Classification for Deep Neural Networks](https://mlsanity.com/paper/1705.08500/) (2017)
- [Predictive Uncertainty Estimation via Prior Networks](https://mlsanity.com/paper/1802.10501/) (2018)
- [A Unifying Review of Deep and Shallow Anomaly Detection](https://mlsanity.com/paper/2009.11732/) (2020)
- [Open-Set Recognition: a Good Closed-Set Classifier is All You Need?](https://mlsanity.com/paper/2110.06207/) (2021)
- [SelectiveNet: A Deep Neural Network with an Integrated Reject Option](https://mlsanity.com/paper/1901.09192/) (2019)
- [Exploring the Limits of Out-of-Distribution Detection](https://mlsanity.com/paper/2106.03004/) (2021)
- [Learning Confidence for Out-of-Distribution Detection in Neural Networks](https://mlsanity.com/paper/1802.04865/) (2018)
- [Selective Question Answering under Domain Shift](https://mlsanity.com/paper/2006.09462/) (2020)
- [To Trust Or Not To Trust A Classifier](https://mlsanity.com/paper/1805.11783/) (2018)
- [A Simple Fix to Mahalanobis Distance for Improving Near-OOD Detection](https://mlsanity.com/paper/2106.09022/) (2021)
- [Addressing Failure Prediction by Learning Model Confidence](https://mlsanity.com/paper/1910.04851/) (2019)
- [Contrastive Training for Improved Out-of-Distribution Detection](https://mlsanity.com/paper/2007.05566/) (2020)
- [Understanding Measures of Uncertainty for Adversarial Example Detection](https://mlsanity.com/paper/1803.08533/) (2018)
- [A Fine-Grained Analysis on Distribution Shift](https://mlsanity.com/paper/2110.11328/) (2021)
- [BREEDS: Benchmarks for Subpopulation Shift](https://mlsanity.com/paper/2008.04859/) (2020)

## Built on by (most cited first, 30 of 52)

- [Document Understanding Dataset and Evaluation (DUDE)](https://mlsanity.com/paper/2305.08455/) (2023)
- [OpenMix: Exploring Outlier Samples for Misclassification Detection](https://mlsanity.com/paper/2303.17093/) (2023)
- [Overcoming Common Flaws in the Evaluation of Selective Classification Systems](https://mlsanity.com/paper/2407.01032/) (2024)
- [Navigating the Pitfalls of Active Learning Evaluation: A Systematic Framework for Meaningful Performance Assessment](https://mlsanity.com/paper/2301.10625/) (2023)
- [Out-of-Distribution Detection: A Task-Oriented Survey of Recent Advances](https://mlsanity.com/paper/2409.11884/) (2024)
- [ValUES: A Framework for Systematic Validation of Uncertainty Estimation in Semantic Segmentation](https://mlsanity.com/paper/2401.08501/) (2024)
- [ImageNet-OOD: Deciphering Modern Out-of-Distribution Detection Algorithms](https://mlsanity.com/paper/2310.01755/) (2023)
- [Deep Learning based Visually Rich Document Content Understanding: A Survey](https://mlsanity.com/paper/2408.01287/) (2024)
- [Window-Based Early-Exit Cascades for Uncertainty Estimation: When Deep Ensembles are More Efficient than Single Models](https://mlsanity.com/paper/2303.08010/) (2023)
- [Benchmarking common uncertainty estimation methods with histopathological images under domain shift and label noise](https://mlsanity.com/paper/2301.01054/) (2023)
- [Local-Prompt: Extensible Local Prompts for Few-Shot Out-of-Distribution Detection](https://mlsanity.com/paper/2409.04796/) (2024)
- [A noisy elephant in the room: Is your out-of-distribution detector robust to label noise?](https://mlsanity.com/paper/2404.01775/) (2024)
- [Comparative Benchmarking of Failure Detection Methods in Medical Image Segmentation: Unveiling the Role of Confidence Aggregation](https://mlsanity.com/paper/2406.03323/) (2024)
- [Great Models Think Alike: Improving Model Reliability via Inter-Model Latent Agreement](https://mlsanity.com/paper/2305.01481/) (2023)
- [Towards Understanding Why Label Smoothing Degrades Selective Classification and How to Fix It](https://mlsanity.com/paper/2403.14715/) (2024)
- [Beyond Document Page Classification: Design, Datasets, and Challenges](https://mlsanity.com/paper/2308.12896/) (2023)
- [Unified Classification and Rejection: A One-versus-All Framework](https://mlsanity.com/paper/2311.13355/) (2023)
- [Conservative Prediction via Data-Driven Confidence Minimization](https://mlsanity.com/paper/2306.04974/) (2023)
- [Are We Ready for Out-of-Distribution Detection in Digital Pathology?](https://mlsanity.com/paper/2407.13708/) (2024)
- [Interpretable Failure Detection with Human-Level Concepts](https://mlsanity.com/paper/2502.05275/) (2025)
- [Learning Credal Ensembles via Distributionally Robust Optimization](https://mlsanity.com/paper/2602.08470/) (2026)
- [Typicalness-Aware Learning for Failure Detection](https://mlsanity.com/paper/2411.01981/) (2024)
- [Pitfalls of Conformal Predictions for Medical Image Classification](https://mlsanity.com/paper/2506.18162/) (2025)
- [Understanding Silent Failures in Medical Image Classification](https://mlsanity.com/paper/2307.14729/) (2023)
- [Jointly Exploring Client Drift and Catastrophic Forgetting in Dynamic Learning](https://mlsanity.com/paper/2309.00688/) (2023)
- [The Cost of Reasoning: Chain-of-Thought Induces Overconfidence in Vision-Language Models](https://mlsanity.com/paper/2603.16728/) (2026)
- [QUTE: Quantifying Uncertainty in TinyML with Early-exit-assisted ensembles for model-monitoring](https://mlsanity.com/paper/2404.12599/) (2024)
- [Why context matters in VQA and Reasoning: Semantic interventions for VLM input modalities](https://mlsanity.com/paper/2410.01690/) (2024)
- [Rejection via Learning Density Ratios](https://mlsanity.com/paper/2405.18686/) (2024)
- [A Rate-Distortion View of Uncertainty Quantification](https://mlsanity.com/paper/2406.10775/) (2024)

## Abstract (arXiv)

Reliable application of machine learning-based decision systems in the wild is one of the major challenges currently investigated by the field. A large portion of established approaches aims to detect erroneous predictions by means of assigning confidence scores. This confidence may be obtained by either quantifying the model's predictive uncertainty, learning explicit scoring functions, or assessing whether the input is in line with the training distribution. Curiously, while these approaches all state to address the same eventual goal of detecting failures of a classifier upon real-life application, they currently constitute largely separated research fields with individual evaluation protocols, which either exclude a substantial part of relevant methods or ignore large parts of relevant failure sources. In this work, we systematically reveal current pitfalls caused by these inconsistencies and derive requirements for a holistic and realistic evaluation of failure detection. To demonstrate the relevance of this unified perspective, we present a large-scale empirical study for the first time enabling benchmarking confidence scoring functions w.r.t all relevant methods and failure sources. The revelation of a simple softmax response baseline as the overall best performing method underlines the drastic shortcomings of current evaluation in the abundance of publicized research on confidence scoring. Code and trained models are at https://github.com/IML-DKFZ/fd-shifts.