# Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of Large Language Models

arXiv: https://arxiv.org/abs/2503.02623 · HTML: https://mlsanity.com/paper/2503.02623/

## Key facts

- Cited by 52 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 51 references that are in the map.
- First posted on arXiv 4 Mar 2025.

## Builds on (most cited first, 30 of 39)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation](https://mlsanity.com/paper/2302.09664/) (2023)
- [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023)
- [SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models](https://mlsanity.com/paper/2303.08896/) (2023)
- [Teaching Models to Express Their Uncertainty in Words](https://mlsanity.com/paper/2205.14334/) (2022)
- [Towards Generalist Biomedical AI](https://mlsanity.com/paper/2307.14334/) (2023)
- [A Survey of Confidence Estimation and Calibration in Large Language Models](https://mlsanity.com/paper/2311.08298/) (2023)
- [Quark: Controllable Text Generation with Reinforced Unlearning](https://mlsanity.com/paper/2205.13636/) (2022)
- [Reducing conversational agents' overconfidence through linguistic calibration](https://mlsanity.com/paper/2012.14983/) (2020)
- [Grounding Large Language Models in Interactive Environments with Online Reinforcement Learning](https://mlsanity.com/paper/2302.02662/) (2023)
- [Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Models](https://mlsanity.com/paper/2307.01379/) (2023)
- [Taming Overconfidence in LLMs: Reward Calibration in RLHF](https://mlsanity.com/paper/2410.09724/) (2024)
- [R-Tuning: Instructing Large Language Models to Say `I Don't Know'](https://mlsanity.com/paper/2311.09677/) (2023)
- [Unfamiliar Finetuning Examples Control How Language Models Hallucinate](https://mlsanity.com/paper/2403.05612/) (2024)
- [Calibration in Deep Learning: A Survey of the State-of-the-Art](https://mlsanity.com/paper/2308.01222/) (2023)
- [Alignment for Honesty](https://mlsanity.com/paper/2312.07000/) (2023)
- [Rejection Improves Reliability: Training LLMs to Refuse Unknown Questions Using RL from Knowledge Feedback](https://mlsanity.com/paper/2403.18349/) (2024)
- [Calibrating Large Language Models Using Their Generations Only](https://mlsanity.com/paper/2403.05973/) (2024)
- [Can AI Assistants Know What They Don't Know?](https://mlsanity.com/paper/2401.13275/) (2024)

## Built on by (most cited first, 30 of 52)

- [TruthRL: Incentivizing Truthful LLMs via Reinforcement Learning](https://mlsanity.com/paper/2509.25760/) (2025)
- [Mitigating LLM Hallucination via Behaviorally Calibrated Reinforcement Learning](https://mlsanity.com/paper/2512.19920/) (2025)
- [Demystifying Multi-Agent Debate: The Role of Confidence and Diversity](https://mlsanity.com/paper/2601.19921/) (2026)
- [Post-Training Large Language Models via Reinforcement Learning from Self-Feedback](https://mlsanity.com/paper/2507.21931/) (2025)
- [The Illusion of Certainty: Decoupling Capability and Calibration in On-Policy Distillation](https://mlsanity.com/paper/2604.16830/) (2026)
- [BARREL: Boundary-Aware Reasoning for Factual and Reliable LRMs](https://mlsanity.com/paper/2505.13529/) (2025)
- [Closing the Confidence-Faithfulness Gap in Large Language Models](https://mlsanity.com/paper/2603.25052/) (2026)
- [VL-Calibration: Decoupled Confidence Calibration for Large Vision-Language Models Reasoning](https://mlsanity.com/paper/2604.09529/) (2026)
- [Open-Jev Judgments on CallScreenBench: Calibrated One-Pass Scam Screening with a Small Language Model](https://mlsanity.com/paper/2609.23959/) (2026)
- [Counterfactual Graph for Multi-Agent LLM Calibration](https://mlsanity.com/paper/2605.30653/) (2026)
- [Self-Evaluating LLMs for Multi-Step Tasks: Stepwise Confidence Estimation for Failure Detection](https://mlsanity.com/paper/2511.07364/) (2025)
- [ConfClip: Confidence-Weighted and Clipped Reward for Reinforcement Learning in LLMs](https://mlsanity.com/paper/2509.17730/) (2025)
- [AdvJudge-Zero: Binary Decision Flips in LLM-as-a-Judge via Adversarial Control Tokens](https://mlsanity.com/paper/2512.17375/) (2025)
- [Confidence-Calibrated Small-Large Language Model Collaboration for Cost-Efficient Reasoning](https://mlsanity.com/paper/2603.03752/) (2026)
- [Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMs](https://mlsanity.com/paper/2606.32032/) (2026)
- [Judging with Confidence: Calibrating Autoraters to Preference Distributions](https://mlsanity.com/paper/2510.00263/) (2025)
- [Unsupervised Confidence Calibration for Reasoning LLMs from a Single Generation](https://mlsanity.com/paper/2604.19444/) (2026)
- [Closing the Reflection Gap: A Free Calibration Bonus for Agentic RL](https://mlsanity.com/paper/2606.14211/) (2026)
- [Just how sure are you? Improving Verbalized Uncertainty Calibration in Medical VQA](https://mlsanity.com/paper/2606.27023/) (2026)
- [Typed Decision Models: An Early Evidence Audit and Evaluation Checklist](https://mlsanity.com/paper/2609.32160/) (2026)
- [REHEARSE: Experiential Rehearsal for Verbal Confidence Calibration in Large Language Models](https://mlsanity.com/paper/2508.14390/) (2025)
- [Conf-Profile: A Confidence-Driven Reasoning Paradigm for Label-Free User Profiling](https://mlsanity.com/paper/2509.18864/) (2025)
- [Epistemic Regret Minimization: Label-Free Causal Critique Beyond Outcome Reward](https://mlsanity.com/paper/2602.11675/) (2026)
- [Process Supervision of Confidence Margin for Calibrated LLM Reasoning](https://mlsanity.com/paper/2604.23333/) (2026)
- [On the effectiveness of reward functions in reinforcement learning for confidence calibration of large language models](https://mlsanity.com/paper/2607.04332/) (2026)
- [Dist2ill: Distributional Distillation for One-Pass Uncertainty Estimation in Large Language Models](https://mlsanity.com/paper/2505.11731/) (2025)
- [From Entropy to Calibrated Uncertainty: Training Language Models to Reason About Uncertainty](https://mlsanity.com/paper/2603.06317/) (2026)
- [Self-Calibrating Language Models via Test-Time Discriminative Distillation](https://mlsanity.com/paper/2604.09624/) (2026)
- [Calibrating LLMs with Semantic-level Reward](https://mlsanity.com/paper/2605.15588/) (2026)
- [Reinforcement Learning over Predictive Distributions for LLM Regression](https://mlsanity.com/paper/2605.20740/) (2026)

## Abstract (arXiv)

A safe and trustworthy use of Large Language Models (LLMs) requires an accurate expression of confidence in their answers. We propose a novel Reinforcement Learning approach that allows to directly fine-tune LLMs to express calibrated confidence estimates alongside their answers to factual questions. Our method optimizes a reward based on the logarithmic scoring rule, explicitly penalizing both over- and under-confidence. This encourages the model to align its confidence estimates with the actual predictive accuracy. The optimal policy under our reward design would result in perfectly calibrated confidence expressions. Unlike prior approaches that decouple confidence estimation from response generation, our method integrates confidence calibration seamlessly into the generative process of the LLM. Empirically, we demonstrate that models trained with our approach exhibit substantially improved calibration and generalize to unseen tasks without further fine-tuning, suggesting the emergence of general confidence awareness.