mlsanityData to 7 Oct 2026

Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs

arXiv 2306.13063 ↗First posted 22 Jun 2023cs.CLcited by 946

Not on the map. Why?

Cited by 147 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Empowering large language models to accurately express confidence in their answers is essential for trustworthy decision-making. Previous confidence elicitation methods, which primarily rely on white-box access to internal model information or model fine-tuning, have become less suitable for LLMs, especially closed-source commercial APIs. This leads to a growing need to explore the untapped area of black-box approaches for LLM uncertainty estimation. To better break down the problem, we define a systematic framework with three components: prompting strategies for eliciting verbalized confidence, sampling methods for generating multiple responses, and aggregation techniques for computing consistency. We then benchmark these methods on two key tasks-confidence calibration and failure prediction-across five types of datasets (e.g., commonsense and arithmetic reasoning) and five widely-used LLMs including GPT-4 and LLaMA 2 Chat. Our analysis uncovers several key insights: 1) LLMs, when verbalizing their confidence, tend to be overconfident, potentially imitating human patterns of expressing confidence. 2) As model capability scales up, both calibration and failure prediction performance improve. 3) Employing our proposed strategies, such as human-inspired prompts, consistency among multiple responses, and better aggregation strategies can help mitigate this overconfidence from various perspectives. 4) Comparisons with white-box methods indicate that while white-box methods perform better, the gap is narrow, e.g., 0.522 to 0.605 in AUROC. Despite these advancements, none of these techniques consistently outperform others, and all investigated methods struggle in challenging tasks, such as those requiring professional knowledge, indicating significant scope for improvement. We believe this study can serve as a strong baseline and provide insights for eliciting confidence in black-box LLMs.

Key facts. Cited by 946 papers in the map (data through 7 Oct 2026). Builds on 35 of its 52 references that are in the map. First posted on arXiv 22 Jun 2023.

Builds on

35 of 52 references.

References · 35

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2017On Calibration of Modern Neural Networkscited by 4,860
  8. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  9. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  10. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  11. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  12. 2022Language Models (Mostly) Know What They Knowcited by 1,783
23 more
  1. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 861
  2. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  3. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  4. 2021Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategiescited by 736
  5. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  6. 2021A Survey of Uncertainty in Deep Neural Networkscited by 675
  7. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 550
  8. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  9. 2018Accurate Uncertainties for Deep Learning Using Calibrated Regressioncited by 417
  10. 2021Revisiting the Calibration of Modern Neural Networkscited by 387
  11. 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 209
  12. 2022Out-of-Distribution Detection and Selective Generation for Conditional Language Modelscited by 193
  13. 2020Mix-n-Match: Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learningcited by 190
  14. 2021On Hallucination and Predictive Uncertainty in Conditional Language Generationcited by 176
  15. 2020Large-scale Robust Deep AUC Maximization: A New Surrogate Loss and Empirical Studies on Medical Image Classificationcited by 84
  16. 2022A Close Look into the Calibration of Pre-trained Language Modelscited by 57
  17. 2020Towards Trustworthy Predictions from Deep Neural Networks with Fast Adversarial Calibrationcited by 34
  18. 2021Calibrated and Sharp Uncertainties in Deep Learning via Density Estimationcited by 33
  19. 2023Proximity-Informed Calibration for Deep Neural Networkscited by 18
  20. 2023Great Models Think Alike: Improving Model Reliability via Inter-Model Latent Agreementcited by 10
  21. 2022Birds of a Feather Trust Together: Knowing When to Trust a Classifier via Adaptive Neighborhood Aggregationcited by 7
  22. 2019Learning to Count Objects with Few Exemplar Annotationscited by 4
  23. 2013How to Elicit Many Probabilitiescited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 43; 2024: 200; 2025: 298; 2026: 405.

946 papers.

Sort:

By area:

  1. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  2. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  3. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 450
  4. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  5. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  6. 2023Explainability for Large Language Models: A Surveycited by 331
  7. 2023ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMscited by 297
  8. 2023A Survey on Multimodal Large Language Models for Autonomous Drivingcited by 288
  9. 2023A Dynamic LLM-Powered Agent Network for Task-Oriented Agent Collaborationcited by 281
  10. 2023RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Modelscited by 272
  11. 2023Exploring the Potential of Large Language Models (LLMs) in Learning on Graphscited by 271
  12. 2024To CoT or not to CoT? Chain-of-thought helps mainly on math and symbolic reasoningcited by 257
38 more
  1. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 253
  2. 2023Can LLM-Generated Misinformation Be Detected?cited by 173
  3. 2023RoleLLM: Benchmarking, Eliciting, and Enhancing Role-Playing Abilities of Large Language Modelscited by 173
  4. 2025Reasoning Models Know When They're Right: Probing Hidden States for Self-Verificationcited by 154
  5. 2024Large Legal Fictions: Profiling Legal Hallucinations in Large Language Modelscited by 144
  6. 2023Large Language Model Cascades with Mixture of Thoughts Representations for Cost-efficient Reasoningcited by 140
  7. 2024LAB-Bench: Measuring Capabilities of Language Models for Biology Researchcited by 138
  8. 2025AbstentionBench: Reasoning LLMs Fail on Unanswerable Questionscited by 129
  9. 2024Benchmarking LLMs via Uncertainty Quantificationcited by 124
  10. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
  11. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 115
  12. 2024Rethinking Interpretability in the Era of Large Language Modelscited by 110
  13. 2024What Large Language Models Know and What People Think They Knowcited by 109
  14. 2025Beyond Binary Rewards: Training LMs to Reason About Their Uncertaintycited by 107
  15. 2024Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertaintycited by 106
  16. 2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 105
  17. 2024SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationalescited by 102
  18. 2024Large Language Models Must Be Taught to Know What They Don't Knowcited by 99
  19. 2024Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreementcited by 99
  20. 2025Maximizing Confidence Alone Improves Reasoningcited by 97
  21. 2024HaloScope: Harnessing Unlabeled LLM Generations for Hallucination Detectioncited by 96
  22. 2025Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areascited by 93
  23. 2025Uncertainty Quantification and Confidence Calibration in Large Language Models: A Surveycited by 93
  24. 2024On Verbalized Confidence Scores for LLMscited by 83
  25. 2024LLM Internal States Reveal Hallucination Risk Faced With a Querycited by 80
  26. 2024LUQ: Long-text Uncertainty Quantification for LLMscited by 79
  27. 2023Self-Evaluation Improves Selective Generation in Large Language Modelscited by 74
  28. 2025Efficient Test-Time Scaling via Self-Calibrationcited by 74
  29. 2025SAFE: Multitask Failure Detection for Vision-Language-Action Modelscited by 74
  30. 2023Label-free Node Classification on Graphs with Large Language Models (LLMS)cited by 72
  31. 2023Alignment for Honestycited by 72
  32. 2024Calibrating Large Language Models with Sample Consistencycited by 72
  33. 2024VL-Uncertainty: Detecting Hallucination in Large Vision-Language Model via Uncertainty Estimationcited by 71
  34. 2024"I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trustcited by 67
  35. 2024Rejection Improves Reliability: Training LLMs to Refuse Unknown Questions Using RL from Knowledge Feedbackcited by 64
  36. 2024Calibrating Large Language Models Using Their Generations Onlycited by 63
  37. 2025Reasoning Models Better Express Their Confidencecited by 63
  38. 2024ConU: Conformal Uncertainty in Large Language Models with Correctness Coverage Guaranteescited by 62

50 most cited of 946 All 946 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Confidence Under the Hood: An Investigation into the Confidence-Probability Alignment in Large Language Models
  2. 2026Not All Uncertainty Is Equal: How Uncertainty Granularity Shapes Human Verification in LLM-Assisted Decision Making ↗
  3. 2024Confidence in the Reasoning of Large Language Models
  4. 2026When Words Fall Short: Iterative Synergy Between Verbalized Reasoning and Hidden Features for LLM Confidence Estimation
  5. 2026Beyond "I Don't Know": Evaluating LLM Self-Awareness in Discriminating Data and Model Uncertainty
  6. 2026Beyond Confidence: Rethinking Self-Assessments for Performance Prediction in LLMs
  7. 2025Can You Trust an LLM with Your Life-Changing Decision? An Investigation into AI High-Stakes Responses
  8. 2025Can LLMs Detect Their Confabulations? Estimating Reliability in Uncertainty-Aware Language Models