mlsanityData to 14 Oct 2026

Taming Overconfidence in LLMs: Reward Calibration in RLHF

arXiv 2410.09724 ↗First posted 13 Oct 2024cs.CLcited by 112

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Language model calibration refers to the alignment between the confidence of the model and the actual performance of its responses. While previous studies point out the overconfidence phenomenon in Large Language Models (LLMs) and show that LLMs trained with Reinforcement Learning from Human Feedback (RLHF) are overconfident with a more sharpened output probability, in this study, we reveal that RLHF tends to lead models to express verbalized overconfidence in their own responses. We investigate the underlying cause of this overconfidence and demonstrate that reward models used for Proximal Policy Optimization (PPO) exhibit inherent biases towards high-confidence scores regardless of the actual quality of responses. Building upon this insight, we propose two PPO variants: PPO-M: PPO with Calibrated Reward Modeling and PPO-C: PPO with Calibrated Reward Calculation. PPO-M integrates explicit confidence scores in reward model training, which calibrates reward models to better capture the alignment between response quality and verbalized confidence. PPO-C adjusts the reward score during PPO based on the difference between the current reward and the exponential average of past rewards. Both PPO-M and PPO-C can be seamlessly integrated into the current PPO pipeline and do not require additional golden labels. We evaluate our methods on both Llama3-8B and Mistral-7B across six diverse datasets including multiple-choice and open-ended generation. Experimental results demonstrate that both of our methods can reduce calibration error and maintain performance comparable to standard PPO. We further show that they could preserve model capabilities in open-ended conversational settings.

Key facts. Cited by 112 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 59 of its 70 references that are in the map. First posted on arXiv 13 Oct 2024.

Builds on

59 of 70 references.

Closest ancestors · 13

Same direction, newest first

  1. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts18 Jun
  2. 2024HelpSteer2: Open-source dataset for training top-performing reward models12 Jun
  3. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizer26 May
  4. 2024RLHF Workflow: From Reward Modeling to Online RLHF13 May
  5. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling11 Jan
  6. 2023SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF9 Oct
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  8. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedback17 May
  9. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  10. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tears11 Apr
  11. 2022Models of human preference for learning reward functions5 Jun
  12. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
1 more
  1. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 46

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  7. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  8. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  10. 2017On Calibration of Modern Neural Networkscited by 4,960
  11. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  12. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
34 more
  1. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  2. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
  3. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  4. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  5. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  6. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  7. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  8. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  9. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
  10. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  11. 2017Crowdsourcing Multiple Choice Science Questionscited by 765
  12. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  13. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  14. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  15. 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 462
  16. 2021Revisiting the Calibration of Modern Neural Networkscited by 390
  17. 2020Calibrating Deep Neural Networks using Focal Losscited by 357
  18. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
  19. 2023Preference Ranking Optimization for Human Alignmentcited by 285
  20. 2023MathPrompter: Mathematical Reasoning using Large Language Modelscited by 236
  21. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 229
  22. 2020Mix-n-Match: Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learningcited by 190
  23. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 148
  24. 2024SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationalescited by 105
  25. 2022Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysiscited by 104
  26. 2024OffsetBias: Leveraging Debiased Data for Tuning Evaluatorscited by 94
  27. 2024Stable LM 2 1.6B Technical Reportcited by 90
  28. 2024CodeUltraFeedback: An LLM-as-a-Judge Dataset for Aligning Large Language Models to Coding Preferencescited by 57
  29. 2024When to Trust LLMs: Aligning Confidence with Response Qualitycited by 43
  30. 2022On the Sensitivity of Reward Inference to Misspecified Human Modelscited by 25
  31. 2024AutoPRM: Automating Procedural Supervision for Multi-Step Reasoning via Controllable Question Decompositioncited by 24
  32. 2024Enhancing Confidence Expression in Large Language Models Through Learning from Past Experiencecited by 21
  33. 2023Great Models Think Alike: Improving Model Reliability via Inter-Model Latent Agreementcited by 11
  34. 2024Reconfidencing LLMs from the Grouping Loss Perspectivecited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 34; 2026: 74.

112 papers.

Sort:

By area:

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  2. 2025Beyond Binary Rewards: Training LMs to Reason About Their Uncertaintycited by 117
  3. 2025Efficient Test-Time Scaling via Self-Calibrationcited by 75
  4. 2025QuestBench: Can LLMs ask the right question to acquire information in reasoning tasks?cited by 63
  5. 2025Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of Large Language Modelscited by 52
  6. 2025Mind the Confidence Gap: Overconfidence, Calibration, and Distractor Effects in Large Language Modelscited by 40
  7. 2025Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approachcited by 25
  8. 2025Do Language Models Mirror Human Confidence? Exploring Psychological Insights to Address Overconfidence in LLMscited by 19
  9. 2025Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missingcited by 14
  10. 2025Calibrating Verbalized Confidence with Self-Generated Distractorscited by 14
  11. 2026Decoupling Reasoning and Confidence: Resurrecting Calibration in Reinforcement Learning from Verifiable Rewardscited by 14
  12. 2024AI-Driven Day-to-Day Route Choicecited by 13
38 more
  1. 2024Sail into the Headwind: Alignment via Robust Rewards and Dynamic Labels against Reward Hackingcited by 13
  2. 2025Large Language Models Do Multi-Label Classification Differentlycited by 13
  3. 2025Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewardscited by 12
  4. 2026Evaluating LLMs in Finance Requires Explicit Bias Considerationcited by 11
  5. 2025When Two LLMs Debate, Both Think They'll Wincited by 10
  6. 2025ADVICE: Answer-Dependent Verbalized Confidence Estimationcited by 10
  7. 2025Do Large Language Models Know What They Are Capable Of?cited by 10
  8. 2026Agent-BRACE: Decoupling Beliefs from Actions in Long-Horizon Tasks via Verbalized State Uncertaintycited by 10
  9. 2025C-LoRA: Contextual Low-Rank Adaptation for Uncertainty Estimation in Large Language Modelscited by 9
  10. 2025Calibration Across Layers: Understanding Calibration Evolution in LLMscited by 9
  11. 2026Consensus is Not Verification: Why Crowd Wisdom Strategies Fail for LLM Truthfulnesscited by 9
  12. 2026The Illusion of Certainty: Decoupling Capability and Calibration in On-Policy Distillationcited by 9
  13. 2025Can Large Language Models Match the Conclusions of Systematic Reviews?cited by 8
  14. 2025C2GSPG: Confidence-calibrated Group Sequence Policy Gradient towards Self-aware Reasoningcited by 8
  15. 2026BaseCal: Unsupervised Confidence Calibration via Base Model Signalscited by 8
  16. 2026Know When You're Wrong: Aligning Confidence with Correctness for LLM Error Detectioncited by 8
  17. 2026Closing the Confidence-Faithfulness Gap in Large Language Modelscited by 8
  18. 2024Epistemic Integrity in Large Language Modelscited by 7
  19. 2026BayesBench: Evaluating LLM Belief Trajectories Under Multi-Turn Evidence Accumulationcited by 7
  20. 2025Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignmentcited by 6
  21. 2025Quantitative LLM Judgescited by 6
  22. 2026LLMs Should Express Uncertainty Explicitlycited by 6
  23. 2026Large Language Models Are Overconfident in Their Own Responsescited by 6
  24. 2025BrowseConf: Confidence-Guided Test-Time Scaling for Web Agentscited by 5
  25. 2025Honesty over Accuracy: Trustworthy Language Models through Reinforced Hesitationcited by 5
  26. 2026I-CALM: Incentivizing Confidence-Aware Abstention for LLM Selective Answeringcited by 5
  27. 2026CATTO: Balancing Preferences and Confidence in Language Modelscited by 4
  28. 2026LLMs Exhibit Significantly Lower Uncertainty in Creative Writing Than Professional Writerscited by 4
  29. 2026The Alignment Tax: Response Homogenization in Aligned LLMs and Its Implications for Uncertainty Estimationcited by 4
  30. 2026Wired for Overconfidence: A Mechanistic Perspective on Inflated Verbalized Confidence in LLMscited by 4
  31. 2026Decomposing Factual Sycophancy in Language Models: How Size and Instruction Tuning Shape Robustnesscited by 4
  32. 2026Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMscited by 4
  33. 2025Retro-Expert: Collaborative Reasoning for Interpretable Retrosynthesiscited by 3
  34. 2025Another Turn, Better Output? A Turn-Wise Analysis of Iterative LLM Promptingcited by 3
  35. 2025Judging with Confidence: Calibrating Autoraters to Preference Distributionscited by 3
  36. 2026Balancing Classification and Calibration Performance in Decision-Making LLMs via Calibration Aware Reinforcement Learningcited by 3
  37. 2026One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Modelscited by 3
  38. 2026Ambiguity Collapse by LLMs: A Taxonomy of Epistemic Riskscited by 3

50 most cited of 112 All 112 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Post-hoc Reward Calibration: A Case Study on Length Bias
  2. 2024Bayesian Reward Models for LLM Alignment
  3. 2026On the effectiveness of reward functions in reinforcement learning for confidence calibration of large language models
  4. 2024Improving Reinforcement Learning from Human Feedback Using Contrastive Rewards
  5. 2025Mitigating Reward Over-Optimization in RLHF via Behavior-Supported Regularization
  6. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  7. 2024How to Evaluate Reward Models for RLHF
  8. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective