mlsanityData to 14 Oct 2026

Calibration, Not Answer Selection: Distilling Internal Confidence in Reasoning Models

arXiv 2609.33290 ↗First posted 27 Sep 2026cs.CLcited by 0

On the map: LLM reasoning → Test-time compute and efficient thinking also fits: Prompting and chain-of-thought reasoning

Abstract (arXiv)

Reinforcement learning with binary correctness rewards trains correctness, not calibrated confidence. The confidence that reasoning models verbalize is systematically overconfident, and the problem is not merely one of scale: verbalized confidence tracks how willing a model is to commit to an answer, not how likely the answer is to be right. Post-hoc rescaling therefore fits one distribution but rarely transfers. We look inside the model instead. On factual question answering, a linear probe on the hidden state between the chain of thought and the answer is substantially better calibrated: its expected calibration error is 5 to 38 times lower than that of the verbalized score across four benchmarks and two model families. However, when used to pick among N sampled answers, that same probe nearly ties majority voting yet falls far short of the oracle. Internal states answer "how certain am I" well and "which answer is right" poorly, so the signal should be reported as a confidence rather than used to select answers. As a result, we introduce probe-guided self-distillation (Probe-SD): score a model's own sampled traces with the probe, overwrite the confidence each trace states, and finetune the base checkpoint of the same family, so nothing but the model itself remains at test time. On Qwen3-14B, Probe-SD cuts ECE from 0.178 to 0.024 in-domain and from 0.542 to 0.113 out-of-domain, where it also beats post-hoc recalibration and self-consistency distillation. The resulting confidence is well-calibrated and useful for weighted voting, behaviors previously attributed to online RL, here obtained with supervised finetuning alone.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 56 of its 63 references that are in the map. First posted on arXiv 27 Sep 2026.

Builds on

56 of 63 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026Unsupervised Confidence Calibration for Reasoning LLMs from a Single Generation21 Apr
  2. 2025Trace Length is a Simple Uncertainty Signal in Reasoning Models12 Oct
  3. 2025Reasoning about Uncertainty: Do Reasoning Models Know When They Don't Know?22 Jun
  4. 2025Reasoning Models Better Express Their Confidence20 May
  5. 2025Reasoning Models Know When They're Right: Probing Hidden States for Self-Verification7 Apr
  6. 2022Language Models (Mostly) Know What They Know11 Jul
  7. 2022STaR: Bootstrapping Reasoning With Reasoning28 Mar

Other references · 49

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  6. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  7. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  8. 2024OpenAI o1 System Cardcited by 1,244
  9. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  10. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
  11. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  12. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
37 more
  1. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  2. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 712
  3. 2023The Internal State of an LLM Knows When It's Lyingcited by 659
  4. 2026GLM-5: from Vibe Coding to Agentic Engineeringcited by 587
  5. 2024Measuring short-form factuality in large language modelscited by 373
  6. 2024SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuningcited by 355
  7. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 349
  8. 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 259
  9. 2026Gemma 4 Technical Reportcited by 259
  10. 2024LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinationscited by 244
  11. 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 210
  12. 2021Simple Entity-Centric Questions Challenge Dense Retrieverscited by 140
  13. 2025Beyond Binary Rewards: Training LMs to Reason About Their Uncertaintycited by 117
  14. 2026Ministral 3cited by 114
  15. 2024SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationalescited by 105
  16. 2024Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words?cited by 87
  17. 2024On Verbalized Confidence Scores for LLMscited by 83
  18. 2025SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledgecited by 54
  19. 2025Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of Large Language Modelscited by 52
  20. 2025Calibrating Verbal Uncertainty as a Linear Feature to Reduce Hallucinationscited by 48
  21. 2026Embarrassingly Simple Self-Distillation Improves Code Generationcited by 40
  22. 2025ConfTuner: Training Large Language Models to Express Their Confidence Verballycited by 39
  23. 2025Towards Fully Exploiting LLM Internal States to Enhance Knowledge Boundary Perceptioncited by 27
  24. 2025Revisiting Uncertainty Estimation and Calibration of Large Language Modelscited by 23
  25. 2026Thinking to Recall: How Reasoning Unlocks Parametric Knowledge in LLMscited by 16
  26. 2024LACIE: Listener-Aware Finetuning for Confidence Calibration in Large Language Modelscited by 15
  27. 2024Perceptions of Linguistic Uncertainty by Language Models and Humanscited by 15
  28. 2024Enhancing Language Model Factuality via Activation-Based Confidence Calibration and Guided Decodingcited by 14
  29. 2026Decoupling Reasoning and Confidence: Resurrecting Calibration in Reinforcement Learning from Verifiable Rewardscited by 14
  30. 2025Are the Hidden States Hiding Something? Testing the Limits of Factuality-Encoding Capabilities in LLMscited by 11
  31. 2025Calibrating LLM Judges: Linear Probes for Fast and Reliable Uncertainty Estimationcited by 9
  32. 2026Reported Confidence in LLMs Tracks Commitment More Than Correctnesscited by 8
  33. 2025Inside-Out: Hidden Factual Knowledge in LLMscited by 7
  34. 2026LLMs Should Express Uncertainty Explicitlycited by 6
  35. 2026Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMscited by 4
  36. 2025High Accuracy, Less Talk (HALT): Reliable LLMs through Capability-Aligned Finetuningcited by 2
  37. 2026Calibrating Overconfidence Without Sacrificing Confidence: Probe-Conditioned Head Intervention for LLMscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026On the Pitfalls of Verbalized Confidence Priors for Calibrating Large Reasoning Models
  2. 2026CALIBER: Calibrating Confidence Before and After Reasoning in Language Models
  3. 2025Verbalized Confidence Triggers Self-Verification: Emergent Behavior Without Explicit Reasoning Supervision
  4. 2026ORCE: Order-Aware Alignment of Verbalized Confidence in Large Language Models
  5. 2026Beyond Verbalized Confidence: Calibrating Reasoners with Differentiable Readouts
  6. 2026Post-Training Shifts Confidence: A Three-Stage Analysis of How SFT, RL, and OPD Shape CoT Calibration
  7. 2025Thinking Out Loud: Do Reasoning Models Know When They're Right?
  8. 2026Test-time Calibration Learning for Large Language Model Reasoning