mlsanityData to 14 Oct 2026

Read Your Own Mind: Reasoning Helps Surface Self-Confidence Signals in LLMs

arXiv 2505.23845 ↗First posted 28 May 2025cs.CLcited by 10

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

We study the source of uncertainty in DeepSeek R1-32B by analyzing its self-reported verbal confidence on question answering (QA) tasks. In the default answer-then-confidence setting, the model is regularly over-confident, whereas semantic entropy - obtained by sampling many responses - remains reliable. We hypothesize that this is because of semantic entropy's larger test-time compute, which lets us explore the model's predictive distribution. We show that granting DeepSeek the budget to explore its distribution by forcing a long chain-of-thought before the final answer greatly improves its verbal score effectiveness, even on simple fact-retrieval questions that normally require no reasoning. Furthermore, a separate reader model that sees only the chain can reconstruct very similar confidences, indicating the verbal score might be merely a statistic of the alternatives surfaced during reasoning. Our analysis concludes that reliable uncertainty estimation requires explicit exploration of the generative space, and self-reported confidence is trustworthy only after such exploration.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 23 of its 28 references that are in the map. First posted on arXiv 28 May 2025.

Builds on

23 of 28 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Is That Your Final Answer? Test-Time Scaling Improves Selective Question Answering19 Feb
  2. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters6 Aug

Other references · 21

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  6. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  7. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  8. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  9. 2024Measuring short-form factuality in large language modelscited by 373
  10. 2023Embers of Autoregression: Understanding Large Language Models Through the Problem They are Trained to Solvecited by 178
  11. 2023Sources of Hallucination by Large Language Models on Inference Taskscited by 169
  12. 2024On Verbalized Confidence Scores for LLMscited by 83
9 more
  1. 2022Mutual Information Alleviates Hallucinations in Abstractive Summarizationcited by 68
  2. 2024Linguistic Calibration of Long-Form Generationscited by 60
  3. 2024Calibrating the Confidence of Large Language Models by Eliciting Fidelitycited by 47
  4. 2023Faithfulness-Aware Decoding Strategies for Abstractive Summarizationcited by 31
  5. 2024Fact-and-Reflection (FaR) Improves Confidence Calibration of Large Language Modelscited by 27
  6. 2024Are Large Language Models More Honest in Their Probabilistic or Verbalized Confidence?cited by 23
  7. 2024Confidence in the Reasoning of Large Language Modelscited by 18
  8. 2025Beyond the Last Answer: Your Reasoning Trace Uncovers More than You Thinkcited by 16
  9. 2022Calibration of Machine Reading Systems at Scalecited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 8.

10 papers.

  1. 2025BrokenMath: A Benchmark for Sycophancy in Theorem Proving with LLMscited by 27
  2. 2025Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missingcited by 14
  3. 2026Hallucinations Undermine Trust; Metacognition is a Way Forwardcited by 5
  4. 2026SELFDOUBT: Uncertainty Quantification for Reasoning LLMs via the Hedge-to-Verify Ratiocited by 3
  5. 2026How Uncertainty Estimation Scales with Sampling in Reasoning Modelscited by 1
  6. 2026Future Confidence Distillation in Large Language Modelscited by 1
  7. 2026Enhancing Confidence Estimation in Telco LLMs via Twin-Pass CoT-Ensemblingcited by 0
  8. 2026Operadic consistency: a label-free signal for compositional reasoning failures in LLMscited by 0
  9. 2026Auto-Fill: Learning to Predict Missing Values Accurately with Specialist Language Modelscited by 0
  10. 2026Different Facets of Verbalised Overconfidence: an Interpretability Studycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026How do LLMs Compute Verbal Confidence
  2. 2026LLMs Should Express Uncertainty Explicitly
  3. 2025Confidence Improves Self-Consistency in LLMs
  4. 2025Think Just Enough: Sequence-Level Entropy as a Confidence Signal for LLM Reasoning
  5. 2026Probability is Not Enough: Exploring and Counting Divergent Tokens for Reasoning Uncertainty Quantification in LLMs
  6. 2025Don't Think Twice! Over-Reasoning Impairs Confidence Calibration
  7. 2025Lexical Hints of Accuracy in LLM Reasoning Chains
  8. 2026Also Small Models Can Reasonably Self-Evaluate Their Confidence