mlsanityData to 14 Oct 2026

Confidence Under the Hood: An Investigation into the Confidence-Probability Alignment in Large Language Models

arXiv 2405.16282 ↗First posted 25 May 2024cs.CL cs.AI cs.LGcited by 34

Not on the map. Why?

Abstract (arXiv)

As the use of Large Language Models (LLMs) becomes more widespread, understanding their self-evaluation of confidence in generated responses becomes increasingly important as it is integral to the reliability of the output of these models. We introduce the concept of Confidence-Probability Alignment, that connects an LLM's internal confidence, quantified by token probabilities, to the confidence conveyed in the model's response when explicitly asked about its certainty. Using various datasets and prompting techniques that encourage model introspection, we probe the alignment between models' internal and expressed confidence. These techniques encompass using structured evaluation scales to rate confidence, including answer options when prompting, and eliciting the model's confidence level for outputs it does not recognize as its own. Notably, among the models analyzed, OpenAI's GPT-4 showed the strongest confidence-probability alignment, with an average Spearman's hatρ of 0.42, across a wide range of tasks. Our work contributes to the ongoing efforts to facilitate risk assessment in the application of LLMs and to further our understanding of model trustworthiness.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Builds on 21 of its 36 references that are in the map. First posted on arXiv 25 May 2024.

Builds on

21 of 36 references.

References · 21

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  9. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  10. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  11. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  12. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
9 more
  1. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 692
  2. 2022Large Language Models Can Self-Improvecited by 664
  3. 2019QASC: A Dataset for Question Answering via Sentence Compositioncited by 252
  4. 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 210
  5. 2023ReWOO: Decoupling Reasoning from Observations for Efficient Augmented Language Modelscited by 184
  6. 2023Mindstorms in Natural Language-Based Societies of Mindcited by 107
  7. 2021The Factual Inconsistency Problem in Abstractive Text Summarization: A Surveycited by 79
  8. 2021RiddleSense: Reasoning about Riddle Questions Featuring Linguistic Creativity and Commonsense Knowledgecited by 61
  9. 2023Beyond Confidence: Reliable Models Should Also Consider Atypicalitycited by 25

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 17; 2026: 16.

34 papers.

Sort:

By area:

  1. 2025Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resamplingcited by 33
  2. 2025From Calibration to Collaboration: LLM Uncertainty Quantification Should Be More Human-Centeredcited by 21
  3. 2025Chain-of-Thought Prompting Obscures Hallucination Cues in Large Language Models: An Empirical Evaluationcited by 21
  4. 2025An Empirical Analysis of Uncertainty in Large Language Model Evaluationscited by 20
  5. 2025Do Language Models Mirror Human Confidence? Exploring Psychological Insights to Address Overconfidence in LLMscited by 19
  6. 2024VideoICL: Confidence-based Iterative In-context Learning for Out-of-Distribution Video Understandingcited by 17
  7. 2025MetaFaith: Faithful Natural Language Uncertainty Expression in LLMscited by 16
  8. 2025Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missingcited by 14
  9. 2025Object-Level Verbalized Confidence Calibration in Vision-Language Models via Semantic Perturbationcited by 12
  10. 2025DTS: Enhancing Large Reasoning Models via Decoding Tree Sketchingcited by 6
  11. 2025RGAR: Recurrence Generation-augmented Retrieval for Factual-aware Medical Question Answeringcited by 5
  12. 2026Influential Training Data Retrieval for Explaining Verbalized Confidence of LLMscited by 4
22 more
  1. 2026Instinct vs. Reflection: Unifying Token and Verbalized Confidence in Multimodal Large Modelscited by 4
  2. 2025On the Robustness of Verbal Confidence of LLMs in Adversarial Attackscited by 3
  3. 2025Can VLMs Recall Factual Associations From Visual References?cited by 3
  4. 2026BAPO: Boundary-Aware Policy Optimization for Reliable Agentic Searchcited by 3
  5. 2025Beware of Reasoning Overconfidence: Pitfalls in the Reasoning Process for Multi-solution Taskscited by 2
  6. 2026Reward Modeling for Multi-Agent Orchestrationcited by 2
  7. 2026When Linguistic and Internal Confidence Diverge in Large Language Modelscited by 2
  8. 2025Improving Score Reliability of Multiple Choice Benchmarks with Consistency Evaluation and Altered Answer Choicescited by 1
  9. 2025Direct Confidence Alignment: Aligning Verbalized Confidence with Internal Confidence In Large Language Modelscited by 1
  10. 2026Completing Missing Annotation: Multi-Agent Debate for Accurate and Scalable Relevant Assessment for IR Benchmarkscited by 1
  11. 2026Humans and LLMs Diverge on Probabilistic Inferencescited by 1
  12. 2026Quantitative Introspection in Language Models: Tracking Emotive States Across Conversationcited by 1
  13. 2026Speaking in Self-Assessing Tongues: On the Verbalized Confidence of LLMs in Machine Translationcited by 1
  14. 2025The Effects of Data Augmentation on Confidence Estimation for LLMscited by 0
  15. 2025CHOIR: Collaborative Harmonization fOr Inference Robustnesscited by 0
  16. 2026Fact-Checking with Large Language Models via Probabilistic Certainty and Consistencycited by 0
  17. 2026ViTaB-A: Evaluating Multimodal Large Language Models on Visual Table Attributioncited by 0
  18. 2026AI in Insurance: Adaptive Questionnaires for Improved Risk Profilingcited by 0
  19. 2026Is This Your Final Answer? Cross-Contextual Consistency as a Measure of LLM Credibilitycited by 0
  20. 2026When Confidence Signals Disagree: Local and Global Confidence in Autoregressive Language Modelscited by 0
  21. 2026Foundations of Stochastic Lexical Calculus: Semantic Descent and Random Dynamics on Probability Simplicescited by 0
  22. 2026Verbalized and Internal Probabilities Are Coupled in Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Knowing What You Know Is Not Enough: Large Language Model Confidences Don't Align With Their Actions
  2. 2026An Analysis of Training-Free Self-Reported Confidence in Language Models
  3. 2024Confidence in the Reasoning of Large Language Models
  4. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs
  5. 2026ConfidenceBench: Evaluating Confidence Calibration in Large Language Models
  6. 2026Rethinking Uncertainty Evaluation in Large Language Models
  7. 2024When to Trust LLMs: Aligning Confidence with Response Quality
  8. 2024On Verbalized Confidence Scores for LLMs