mlsanityData to 14 Oct 2026

A Novel Characterization of the Population Area Under the Risk Coverage Curve (AURC) and Rates of Finite Sample Estimators

arXiv 2410.15361 ↗First posted 20 Oct 2024stat.ML cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

The selective classifier (SC) has been proposed for rank based uncertainty thresholding, which could have applications in safety critical areas such as medical diagnostics, autonomous driving, and the justice system. The Area Under the Risk-Coverage Curve (AURC) has emerged as the foremost evaluation metric for assessing the performance of SC systems. In this work, we present a formal statistical formulation of population AURC, presenting an equivalent expression that can be interpreted as a reweighted risk function. Through Monte Carlo methods, we derive empirical AURC plug-in estimators for finite sample scenarios. The weight estimators associated with these plug-in estimators are shown to be consistent, with low bias and tightly bounded mean squared error (MSE). The plug-in estimators are proven to converge at a rate of O(√ln(n)/n) demonstrating statistical consistency. We empirically validate the effectiveness of our estimators through experiments across multiple datasets, model architectures, and confidence score functions (CSFs), demonstrating consistency and effectiveness in fine-tuning AURC performance.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 31 of its 48 references that are in the map. First posted on arXiv 20 Oct 2024.

Builds on

31 of 48 references.

References · 31

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  7. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  8. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  9. 2016Wide Residual Networkscited by 3,070
  10. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  11. 2020Energy-based Out-of-distribution Detectioncited by 1,299
  12. 2017Selective Classification for Deep Neural Networkscited by 852
19 more
  1. 2020A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challengescited by 819
  2. 2019Scaling Out-of-Distribution Detection for Real-World Settingscited by 465
  3. 2017Fairness in Criminal Justice Risk Assessments: The State of the Artcited by 315
  4. 2022Mitigating Neural Network Overconfidence with Logit Normalizationcited by 252
  5. 2023Decomposing Uncertainty for Large Language Models through Input Clarification Ensemblingcited by 114
  6. 2018Bias-Reduced Uncertainty Estimation for Deep Neural Classifierscited by 113
  7. 2020Confidence-Aware Learning for Deep Neural Networkscited by 113
  8. 2021Machine Learning with a Reject Option: A surveycited by 109
  9. 2018Bayesian Uncertainty Estimation for Batch Normalized Deep Networkscited by 70
  10. 2021Optimal strategies for reject option classifierscited by 61
  11. 2024Overcoming Common Flaws in the Evaluation of Selective Classification Systemscited by 37
  12. 2023What Can We Learn From The Selective Prediction And Uncertainty Estimation Performance Of 523 Imagenet Classifierscited by 35
  13. 2023Window-Based Early-Exit Cascades for Uncertainty Estimation: When Deep Ensembles are More Efficient than Single Modelscited by 16
  14. 2023How to Fix a Broken Confidence Estimator: Evaluating Post-hoc Methods for Selective Classification with Deep Neural Networkscited by 16
  15. 2022On the Usefulness of Deep Ensemble Diversity for Out-of-Distribution Detectioncited by 14
  16. 2024Selective Classification Under Distribution Shiftscited by 12
  17. 2024Revisiting Confidence Estimation: Towards Reliable Failure Predictioncited by 10
  18. 2021What classifiers know what they don't?cited by 6
  19. 2023Empirical Optimal Risk to Quantify Model Trustworthiness for Failure Detectioncited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 12.

15 papers.

Sort:

By area:

  1. 2026Entropy Alone is Insufficient for Safe Selective Prediction in LLMscited by 7
  2. 2026When Agents Disagree With Themselves: Behavioral Consistency as an Uncertainty Signal for LLM Agentscited by 3
  3. 2026Decision-Aware Uncertainty Evaluation of Vision-Language Model-Based Early Action Anticipation for Human-Robot Interactioncited by 2
  4. 2026Aligning Language Models with Selective Predictioncited by 2
  5. 2025Trust, or Don't Predict: Introducing the CWSA Family for Confidence-Aware Model Evaluationcited by 1
  6. 2026Prompt-Based Abstention Fails Under Misleading Context: A Controlled Study of Small Frozen RAG Modelscited by 1
  7. 2025Revisiting Reweighted Risk for Calibration: AURC, Focal, and Inverse Focal Losscited by 0
  8. 2025TRUST: Test-time Resource Utilization for Superior Trustworthinesscited by 0
  9. 2026Post-hoc Selective Classification for Reliable Synthetic Image Detectioncited by 0
  10. 2026ECUASn: A family of metrics for principled evaluation of uncertainty-augmented systemscited by 0
  11. 2026Code-MUE: Measuring Code LLMs' Uncertainty through Execution-based Semantic Interaction Graphscited by 0
  12. 2026Target-Checked Reliability Score Refinement for Video Question Answeringcited by 0
3 more
  1. 2026SpatialUQ: Post-Hoc Uncertainty Quantification from Spatial Consistency in Black-Box Vision Modelscited by 0
  2. 2026Can We Trust This Evaluation? Instance-Level Reliability of LLM-as-a-Judgecited by 0
  3. 2026EvoWeb: A Benchmark for Coding Agents on Evolving Web Applicationscited by 0