Confidence Calibration in Large Language Models
Not on the map. Why?
Abstract (arXiv)
We investigate the calibration of large language models' (LLMs') confidence across diverse tasks. The results of our preregistered study show that the current crop of LLMs are, like people, too sure they are right: confidence exceeds accuracy, on average. Importantly, however, this tendency is moderated by a powerful hard-easy effect, wherein overconfidence is greatest on difficult tests; by contrast, easy tests actually show substantial underconfidence. We develop LifeEval, a test for evaluating model calibration across levels of difficulty.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 18 of its 40 references that are in the map. First posted on arXiv 3 Apr 2026.
Builds on
18 of 40 references.
References · 18
- 2017On Calibration of Modern Neural Networkscited by 4,860
- 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
- 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
- 2020A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challengescited by 813
- 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
- 2017Crowdsourcing Multiple Choice Science Questionscited by 742
- 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 701
- 2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 105
- 2025ConfTuner: Training Large Language Models to Express Their Confidence Verballycited by 37
- 2025Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approachcited by 25
- 2025Understanding Model Calibration -- A gentle introduction and visual exploration of calibration and the expected calibration error (ECE)cited by 20
- 2025Do Language Models Mirror Human Confidence? Exploring Psychological Insights to Address Overconfidence in LLMscited by 19
6 more
- 2025The Confidence Paradox: Can LLM Know When It's Wrongcited by 11
- 2025GRACE: A Granular Benchmark for Evaluating Model Calibration against Human Calibrationcited by 5
- 2024Do LLMs Implicitly Determine the Suitable Text Difficulty for Users?cited by 4
- 2024The Evolving Usage of GenAI by Computing Studentscited by 4
- 2025Can LLMs Estimate Cognitive Complexity of Reading Comprehension Items?cited by 4
- 2025Toward Trustworthy Difficulty Assessments: Large Language Models as Judges in Programming and Synthetic Taskscited by 3
Built on by
Citing papers per year: 2026: 1.
1 paper.