Beyond Verbalized Confidence: Calibrating Reasoners with Differentiable Readouts
On the map: LLM reasoning → Reinforcement learning for reasoning
Abstract (arXiv)
Reinforcement learning with verifiable rewards (RLVR) trains reasoning models to produce correct answers, but does not ensure that their stated confidence is calibrated. The resulting models are systematically overconfident. Recent methods train calibration inside the RLVR loop by having the model state a numerical confidence alongside its answer, but they all obtain the confidence by sampling it as text. This choice imposes two costs: a sampled confidence introduces variance and in practice collapses to a handful of distinct values, and sampling makes the confidence non-differentiable, forcing the calibration loss through a scalar reward. We propose CREDO (Confidence REaDOut) to replace sampling with a deterministic readout. While RLVR optimizes correctness, CREDO reads the confidence from a dedicated token pair in the model's output distribution and trains it by differentiable regression. CREDO further turns the trained confidence into a signal for accuracy, weighting rollouts by how far confidence and outcome disagree, so that accuracy and calibration improve together. Across mathematical and code reasoning, CREDO attains the best accuracy and calibration, and the gains extend to abstention and selective prediction.
Key facts. No citing papers in the map yet (data through 13 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 39 of its 52 references that are in the map. First posted on arXiv 28 Sep 2026.
Builds on
39 of 52 references.
Closest ancestors · 10
Same direction, newest first
- 2026Calibration-Aware Policy Optimization for Reasoning LLMs14 Apr
- 2026Decoupling Reasoning and Confidence: Resurrecting Calibration in Reinforcement Learning from Verifiable Rewards10 Mar
- 2026Know What You Know: Metacognitive Entropy Calibration for Verifiable RL Reasoning26 Feb
- 2025LaSeR: Reinforcement Learning with Last-Token Self-Rewarding16 Oct
- 2025C2GSPG: Confidence-calibrated Group Sequence Policy Gradient towards Self-aware Reasoning27 Sep
- 2025Uncalibrated Reasoning: GRPO Induces Overconfidence for Stochastic Outcomes15 Aug
- 2025Beyond Binary Rewards: Training LMs to Reason About Their Uncertainty22 Jul
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
- 2023Let's Verify Step by Step31 May
- 2017Proximal Policy Optimization Algorithms20 Jul
Other references · 29
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,091
- 2025Qwen3 Technical Reportcited by 8,510
- 2021Evaluating Large Language Models Trained on Codecited by 8,205
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,831
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,270
- 2017On Calibration of Modern Neural Networkscited by 4,902
- 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,951
- 2022Language Models (Mostly) Know What They Knowcited by 1,823
- 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,362
- 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 960
- 2017Selective Classification for Deep Neural Networkscited by 825
- 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 810
17 more
- 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 800
- 2022Teaching Models to Express Their Uncertainty in Wordscited by 740
- 2023The Internal State of an LLM Knows When It's Lyingcited by 652
- 2019Monte Carlo Gradient Estimation in Machine Learningcited by 283
- 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 255
- 2024LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinationscited by 236
- 2020Unsupervised Quality Estimation for Neural Machine Translationcited by 211
- 2025AbstentionBench: Reasoning LLMs Fail on Unanswerable Questionscited by 138
- 2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 108
- 2024SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationalescited by 102
- 2024On Verbalized Confidence Scores for LLMscited by 83
- 2024Learning to Route LLMs with Confidence Tokenscited by 67
- 2025Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of Large Language Modelscited by 50
- 2025Reasoning about Uncertainty: Do Reasoning Models Know When They Don't Know?cited by 36
- 2024Improving Uncertainty Estimation through Semantically Diverse Language Generationcited by 34
- 2025LoVeC: Reinforcement Learning for Better Verbalized Confidence in Long-Form Generationscited by 17
- 2026Uncertainty-aware Generative Recommendationcited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026On the Pitfalls of Verbalized Confidence Priors for Calibrating Large Reasoning Models
- 2026RL-ARC: Calibrating Large Reasoning Models via Reasoning-guided Uncertainty
- 2026On the effectiveness of reward functions in reinforcement learning for confidence calibration of large language models
- 2026Process Supervision of Confidence Margin for Calibrated LLM Reasoning
- 2025ConfClip: Confidence-Weighted and Clipped Reward for Reinforcement Learning in LLMs
- 2026Overconfident Errors Need Stronger Correction: Asymmetric Confidence Penalties for Reinforcement Learning
- 2026Scaling with Confidence: Calibrating Confidence of LLMs for Adaptive Test Time Scaling
- 2026Calibration, Not Answer Selection: Distilling Internal Confidence in Reasoning Models