Calibration Is Not Enough: Evaluating Confidence Estimation Under Language Variations
Not on the map. Why?
Abstract (arXiv)
Confidence estimation (CE) indicates how reliable the answers of large language models are and impacts user trust and decision-making. Existing evaluations mainly concern the alignment between confidence and correctness, but ignore the variability of language: confidence estimates should remain consistent under semantically equivalent prompts or answer variations, while changing when answer meaning differs, as this may indicate a change in correctness. Therefore, we introduce a novel evaluation framework based on three complementary properties: robustness to prompt perturbations, stability across semantically equivalent answers, and sensitivity to semantically different answers. We show that these metrics are largely independent from existing CE metrics, and that common CE methods often fail on them: while most methods achieve high robustness and stability, they struggle to distinguish semantically different answers, potentially because they do not effectively leverage generation-side information. Overall, our framework exposes overlooked limitations of current CE evaluations and provides guidance for selecting confidence estimators for real-world applications.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 53 of its 86 references that are in the map. First posted on arXiv 12 Jan 2026.
Builds on
53 of 86 references.
References · 53
- 2017On Calibration of Modern Neural Networkscited by 4,860
- 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
- 2022Language Models (Mostly) Know What They Knowcited by 1,783
- 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
- 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
- 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
- 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
- 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
- 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
- 2017Crowdsourcing Multiple Choice Science Questionscited by 742
- 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
- 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
38 more
- 2023The Internal State of an LLM Knows When It's Lyingcited by 644
- 2019Measuring Calibration in Deep Learningcited by 443
- 2021Revisiting the Calibration of Modern Neural Networkscited by 387
- 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 385
- 2020Calibrating Deep Neural Networks using Focal Losscited by 353
- 2019Verified Uncertainty Calibrationcited by 307
- 2023ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMscited by 297
- 2022Prompting GPT-3 To Be Reliablecited by 258
- 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 253
- 2024LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Taskscited by 236
- 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 235
- 2020Posterior Network: Uncertainty Estimation without OOD Samples via Density-Based Pseudo-Countscited by 163
- 2019Practical Deep Learning with Bayesian Principlescited by 158
- 2024ProSA: Assessing and Understanding the Prompt Sensitivity of LLMscited by 145
- 2023Quantifying Uncertainty in Answers from any Language Model and Enhancing their Trustworthinesscited by 121
- 2024Language Model Cascades: Token-level uncertainty and beyondcited by 115
- 2024Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertaintycited by 106
- 2024Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreementcited by 99
- 2023On the Calibration of Large Language Models and Alignmentcited by 72
- 2024Calibrating Large Language Models Using Their Generations Onlycited by 63
- 2023Uncertainty in Natural Language Generation: From Theory to Applicationscited by 61
- 2023Smooth ECE: Principled Reliability Diagrams via Kernel Smoothingcited by 59
- 2022A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awarenesscited by 49
- 2024Calibrating the Confidence of Large Language Models by Eliciting Fidelitycited by 47
- 2023LitCab: Lightweight Language Model Calibration over Short- and Long-form Responsescited by 44
- 2024Quantifying and Mitigating Unimodal Biases in Multimodal Large Language Models: A Causal Perspectivecited by 39
- 2024Bayesian Reward Models for LLM Alignmentcited by 35
- 2022Calibrated Interpretation: Confidence Estimation in Semantic Parsingcited by 33
- 2024Exploring Precision and Recall to assess the quality and diversity of LLMscited by 27
- 2024Confidence Calibration and Rationalization for LLMs via Multi-Agent Deliberationcited by 26
- 2022Exploring Predictive Uncertainty and Calibration in NLP: A Study on the Impact of Method & Data Scarcitycited by 24
- 2024Are Large Language Models More Honest in Their Probabilistic or Verbalized Confidence?cited by 23
- 2025A Head to Predict and a Head to Question: Pre-trained Uncertainty Quantification Heads for Hallucination Detection in LLM Outputscited by 20
- 2022On the Calibration of Massively Multilingual Language Modelscited by 18
- 2024Black-box Uncertainty Quantification Method for LLM-as-a-Judgecited by 18
- 2025Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missingcited by 14
- 2025Influences on LLM Calibration: A Study of Response Agreement, Loss Functions, and Prompt Stylescited by 12
- 2025Will It Still Be True Tomorrow? Multilingual Evergreen Question Classification to Improve Trustworthy QAcited by 5
50 most cited of 53 All 53 (JSON)
Built on by
Citing papers per year: 2026: 3.
3 papers.