Your Model is Overconfident, and Other Lies We Tell Ourselves
Not on the map. Why?
Abstract (arXiv)
The difficulty intrinsic to a given example, rooted in its inherent ambiguity, is a key yet often overlooked factor in evaluating neural NLP models. We investigate the interplay and divergence among various metrics for assessing intrinsic difficulty, including annotator dissensus, training dynamics, and model confidence. Through a comprehensive analysis using 29 models on three datasets, we reveal that while correlations exist among these metrics, their relationships are neither linear nor monotonic. By disentangling these dimensions of uncertainty, we aim to refine our understanding of data complexity and its implications for evaluating and improving NLP models.
Key facts. Cited by 1 paper in the map (data through 13 Oct 2026). Builds on 21 of its 54 references that are in the map. First posted on arXiv 3 Mar 2025.
Builds on
21 of 54 references.
References · 21
- 2017On Calibration of Modern Neural Networkscited by 4,902
- 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,676
- 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,340
- 2015A large annotated corpus for learning natural language inferencecited by 1,841
- 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,761
- 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,546
- 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 910
- 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 786
- 2018An Empirical Study of Example Forgetting during Deep Neural Network Learningcited by 675
- 2023The Falcon Series of Open Language Modelscited by 533
- 2016Least Ambiguous Set-Valued Classifiers with Bounded Error Levelscited by 272
- 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 255
9 more
- 2019Human uncertainty makes classification more robustcited by 231
- 2021Deep Learning Through the Lens of Example Difficultycited by 141
- 2018Curriculum Learning by Transfer Learning: Theory and Experiments with Deep Networkscited by 113
- 2018Quantifying Uncertainties in Natural Language Processing Taskscited by 79
- 2018How Complex is your classification problem? A survey on measuring classification complexitycited by 66
- 2020DynaSent: A Dynamic Benchmark for Sentiment Analysiscited by 63
- 2023Uncertainty in Natural Language Generation: From Theory to Applicationscited by 61
- 2021Revisiting Uncertainty-based Query Strategies for Active Learning with Transformerscited by 59
- 2023Uncertainty in Natural Language Processing: Sources, Quantification, and Applicationscited by 44
Built on by
Citing papers per year: 2026: 1.
1 paper.