mlsanityData to 7 Oct 2026

Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of Large Language Models

arXiv 2503.02623 ↗First posted 4 Mar 2025cs.CL cs.AIcited by 48

Not on the map. Why?

Abstract (arXiv)

A safe and trustworthy use of Large Language Models (LLMs) requires an accurate expression of confidence in their answers. We propose a novel Reinforcement Learning approach that allows to directly fine-tune LLMs to express calibrated confidence estimates alongside their answers to factual questions. Our method optimizes a reward based on the logarithmic scoring rule, explicitly penalizing both over- and under-confidence. This encourages the model to align its confidence estimates with the actual predictive accuracy. The optimal policy under our reward design would result in perfectly calibrated confidence expressions. Unlike prior approaches that decouple confidence estimation from response generation, our method integrates confidence calibration seamlessly into the generative process of the LLM. Empirically, we demonstrate that models trained with our approach exhibit substantially improved calibration and generalize to unseen tasks without further fine-tuning, suggesting the emergence of general confidence awareness.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Builds on 39 of its 51 references that are in the map. First posted on arXiv 4 Mar 2025.

Builds on

39 of 51 references.

References · 39

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  8. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  9. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  10. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  11. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  12. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
27 more
  1. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  2. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  3. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
  4. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  5. 2023Towards Generalist Biomedical AIcited by 286
  6. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 253
  7. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 236
  8. 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 209
  9. 2023Grounding Large Language Models in Interactive Environments with Online Reinforcement Learningcited by 193
  10. 2023Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Modelscited by 147
  11. 2023R-Tuning: Instructing Large Language Models to Say `I Don't Know'cited by 106
  12. 2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 105
  13. 2024Unfamiliar Finetuning Examples Control How Language Models Hallucinatecited by 85
  14. 2023Calibration in Deep Learning: A Survey of the State-of-the-Artcited by 72
  15. 2023Alignment for Honestycited by 72
  16. 2024Rejection Improves Reliability: Training LLMs to Refuse Unknown Questions Using RL from Knowledge Feedbackcited by 64
  17. 2024Calibrating Large Language Models Using Their Generations Onlycited by 63
  18. 2024Can AI Assistants Know What They Don't Know?cited by 47
  19. 2024When to Trust LLMs: Aligning Confidence with Response Qualitycited by 42
  20. 2024Teaching Large Language Models to Express Knowledge Boundary from Their Own Signalscited by 32
  21. 2024Enhancing Confidence Expression in Large Language Models Through Learning from Past Experiencecited by 21
  22. 2024CHOPS: CHat with custOmer Profile Systems for Customer Service with LLMscited by 17
  23. 2024LACIE: Listener-Aware Finetuning for Confidence Calibration in Large Language Modelscited by 14
  24. 2024LawLuo: A Multi-Agent Collaborative Framework for Multi-Round Chinese Legal Consultationcited by 14
  25. 2023Dialogue Shaping: Empowering Agents through NPC Interactioncited by 8
  26. 2024Knowledge Verification to Nip Hallucination in the Budcited by 7
  27. 2024MAGDA: Multi-agent guideline-driven diagnostic assistancecited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 11; 2026: 37.

48 papers.

Sort:

By area:

  1. 2025TruthRL: Incentivizing Truthful LLMs via Reinforcement Learningcited by 20
  2. 2026Demystifying Multi-Agent Debate: The Role of Confidence and Diversitycited by 17
  3. 2025Post-Training Large Language Models via Reinforcement Learning from Self-Feedbackcited by 14
  4. 2025Mitigating LLM Hallucination via Behaviorally Calibrated Reinforcement Learningcited by 13
  5. 2025BARREL: Boundary-Aware Reasoning for Factual and Reliable LRMscited by 8
  6. 2026Closing the Confidence-Faithfulness Gap in Large Language Modelscited by 8
  7. 2026VL-Calibration: Decoupled Confidence Calibration for Large Vision-Language Models Reasoningcited by 7
  8. 2026Open-Jev Judgments on CallScreenBench: Calibrated One-Pass Scam Screening with a Small Language Modelcited by 7
  9. 2025Self-Evaluating LLMs for Multi-Step Tasks: Stepwise Confidence Estimation for Failure Detectioncited by 6
  10. 2026The Illusion of Certainty: Decoupling Capability and Calibration in On-Policy Distillationcited by 6
  11. 2026Counterfactual Graph for Multi-Agent LLM Calibrationcited by 5
  12. 2025ConfClip: Confidence-Weighted and Clipped Reward for Reinforcement Learning in LLMscited by 4
36 more
  1. 2026Confidence-Calibrated Small-Large Language Model Collaboration for Cost-Efficient Reasoningcited by 4
  2. 2026Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMscited by 4
  3. 2025Judging with Confidence: Calibrating Autoraters to Preference Distributionscited by 3
  4. 2026Closing the Reflection Gap: A Free Calibration Bonus for Agentic RLcited by 3
  5. 2026Just how sure are you? Improving Verbalized Uncertainty Calibration in Medical VQAcited by 3
  6. 2025REHEARSE: Experiential Rehearsal for Verbal Confidence Calibration in Large Language Modelscited by 2
  7. 2025Conf-Profile: A Confidence-Driven Reasoning Paradigm for Label-Free User Profilingcited by 2
  8. 2025AdvJudge-Zero: Binary Decision Flips in LLM-as-a-Judge via Adversarial Control Tokenscited by 2
  9. 2026Epistemic Regret Minimization: Label-Free Causal Critique Beyond Outcome Rewardcited by 2
  10. 2026Unsupervised Confidence Calibration for Reasoning LLMs from a Single Generationcited by 2
  11. 2026On the effectiveness of reward functions in reinforcement learning for confidence calibration of large language modelscited by 2
  12. 2025Dist2ill: Distributional Distillation for One-Pass Uncertainty Estimation in Large Language Modelscited by 1
  13. 2026From Entropy to Calibrated Uncertainty: Training Language Models to Reason About Uncertaintycited by 1
  14. 2026Self-Calibrating Language Models via Test-Time Discriminative Distillationcited by 1
  15. 2026Process Supervision of Confidence Margin for Calibrated LLM Reasoningcited by 1
  16. 2026Calibrating LLMs with Semantic-level Rewardcited by 1
  17. 2026Reinforcement Learning over Predictive Distributions for LLM Regressioncited by 1
  18. 2026Miner:Mining Intrinsic Mastery for Data-Efficient RL in Large Reasoning Modelscited by 0
  19. 2026Calibrated Confidence Expression for Radiology Report Generationcited by 0
  20. 2026Do Not Imitate, Reinforce: Iterative Classification via Belief Refinementcited by 0
  21. 2026Position: Uncertainty Quantification in LLMs is Just Unsupervised Clusteringcited by 0
  22. 2026Look-Closer-Then-Diagnose: Confidence-Aware Ultrasound VQA via Active Zoomingcited by 0
  23. 2026ConSteer-RL: Steering Reasoning Capabilities in Large Language Models via Confidence-Aware Reinforcement Learningcited by 0
  24. 2026CALIBER: Calibrating Confidence Before and After Reasoning in Language Modelscited by 0
  25. 2026From Critic to Confidence: PPO for Language-Based Quantitative Prediction with Confidence Estimationcited by 0
  26. 2026DualStake: Dual-Path Confidence Calibration in Deep Research Agentscited by 0
  27. 2026The Mirage of Calibrated Confidence: Trajectory-Independence of Verbalized Confidence in Vision-Language Modelscited by 0
  28. 2026Learning to Stop without Learning to Stop: Self-Supervised Confidence Training Improves Reasoning Efficiencycited by 0
  29. 2026Typed Decision Models: An Early Evidence Audit and Evaluation Checklistcited by 0
  30. 2026On the Pitfalls of Verbalized Confidence Priors for Calibrating Large Reasoning Modelscited by 0
  31. 2026Calibration, Not Answer Selection: Distilling Internal Confidence in Reasoning Modelscited by 0
  32. 2026Shared Experience, Separate Learning: Companion Confidence Calibration for LLMscited by 0
  33. 2026When Words Fall Short: Iterative Synergy Between Verbalized Reasoning and Hidden Features for LLM Confidence Estimationcited by 0
  34. 2026Beyond Verbalized Confidence: Calibrating Reasoners with Differentiable Readoutscited by 0
  35. 2026Bongard: Training Machine Intuitioncited by 0
  36. 2026OmniMed-Jev: Calibrating LVLM Confidence for Trustworthy Medical Multimodal Decisions via System Onecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Confidence Before Answering: A Paradigm Shift for Efficient LLM Uncertainty Estimation
  2. 2026Scaling with Confidence: Calibrating Confidence of LLMs for Adaptive Test Time Scaling
  3. 2026ORCE: Order-Aware Alignment of Verbalized Confidence in Large Language Models
  4. 2026The ACUTE Protocol: Operationalizing Language Model Activations for Better Calibration, Utility, and Trust
  5. 2025Beyond Correctness: Confidence-Aware Reward Modeling for Enhancing Large Language Model Reasoning
  6. 2025Verbalized Confidence Triggers Self-Verification: Emergent Behavior Without Explicit Reasoning Supervision
  7. 2026Balancing Classification and Calibration Performance in Decision-Making LLMs via Calibration Aware Reinforcement Learning
  8. 2026ConfidenceBench: Evaluating Confidence Calibration in Large Language Models