mlsanityData to 7 Oct 2026

Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach

arXiv 2505.01997 ↗First posted 4 May 2025cs.LG cs.AI stat.MLcited by 25

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

One of the key technologies for the success of Large Language Models (LLMs) is preference alignment. However, a notable side effect of preference alignment is poor calibration: while the pre-trained models are typically well-calibrated, LLMs tend to become poorly calibrated after alignment with human preferences. In this paper, we investigate why preference alignment affects calibration and how to address this issue. For the first question, we observe that the preference collapse issue in alignment undesirably generalizes to the calibration scenario, causing LLMs to exhibit overconfidence and poor calibration. To address this, we demonstrate the importance of fine-tuning with domain-specific knowledge to alleviate the overconfidence issue. To further analyze whether this affects the model's performance, we categorize models into two regimes: calibratable and non-calibratable, defined by bounds of Expected Calibration Error (ECE). In the calibratable regime, we propose a calibration-aware fine-tuning approach to achieve proper calibration without compromising LLMs' performance. However, as models are further fine-tuned for better performance, they enter the non-calibratable regime. For this case, we develop an EM-algorithm-based ECE regularization for the fine-tuning loss to maintain low calibration error. Extensive experiments validate the effectiveness of the proposed methods.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 65 of its 79 references that are in the map. First posted on arXiv 4 May 2025.

Builds on

65 of 79 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024On the Algorithmic Bias of Aligning Large Language Models with RLHF: Preference Collapse and Matching Regularization26 May
  2. 2024Understanding the performance gap between online and offline alignment algorithms14 May
  3. 2024From r to Q^*: Your Language Model is Secretly a Q-Function18 Apr
  4. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study16 Apr
  5. 2024Learn Your Reference Model for Real Good Alignment15 Apr
  6. 2024Asymptotics of Language Model Alignment2 Apr
  7. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  8. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep
  9. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback24 May

Other references · 56

  1. 2017Attention Is All You Needcited by 59,932
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  9. 2017On Calibration of Modern Neural Networkscited by 4,860
  10. 2017Deep reinforcement learning from human preferencescited by 3,846
  11. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  12. 2023Mistral 7Bcited by 3,065
38 more
  1. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  2. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  3. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  4. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  5. 2019When Does Label Smoothing Help?cited by 985
  6. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  7. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  8. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  9. 2019Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibrationcited by 359
  10. 2020Calibrating Deep Neural Networks using Focal Losscited by 353
  11. 2019Verified Uncertainty Calibrationcited by 307
  12. 20242 OLMo 2 Furiouscited by 302
  13. 2023Nash Learning from Human Feedbackcited by 220
  14. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 184
  15. 2020Improving model calibration with accuracy versus uncertainty optimizationcited by 123
  16. 2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 105
  17. 2022Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysiscited by 104
  18. 2024SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationalescited by 102
  19. 2021Soft Calibration Objectives for Neural Networkscited by 89
  20. 2020Distribution-free binary classification: prediction sets, confidence intervals and calibrationcited by 79
  21. 2024Preserving Diversity in Supervised Fine-Tuning of Large Language Modelscited by 68
  22. 2022Better Uncertainty Calibration via Proper Scores for Classification and Beyondcited by 65
  23. 2022Prototypical Calibration for Few-shot Learning of Language Modelscited by 58
  24. 2022A Close Look into the Calibration of Pre-trained Language Modelscited by 57
  25. 2022Sample-dependent Adaptive Temperature Scaling for Improved Calibrationcited by 48
  26. 2021Knowing More About Questions Can Help: Improving Calibration in Question Answeringcited by 47
  27. 2021Distribution-free calibration guarantees for histogram binning without sample splittingcited by 44
  28. 2022A Consistent and Differentiable Lp Canonical Calibration Error Estimatorcited by 44
  29. 2024When to Trust LLMs: Aligning Confidence with Response Qualitycited by 42
  30. 2024Dataset Reset Policy Optimization for RLHFcited by 41
  31. 2022Robust Calibration with Multi-domain Temperature Scalingcited by 39
  32. 2024Thermometer: Towards Universal Calibration for Large Language Modelscited by 38
  33. 2023Policy Optimization in RLHF: The Impact of Out-of-preference Datacited by 35
  34. 2025Statistical Impossibility and Possibility of Aligning LLMs with Human Preferences: From Condorcet Paradox to Nash Equilibriumcited by 22
  35. 2024Enhancing Confidence Expression in Large Language Models Through Learning from Past Experiencecited by 21
  36. 2023Generative Calibration for In-context Learningcited by 20
  37. 2024Magnetic Preference Optimization: Achieving Last-iterate Convergence for Language Model Alignmentcited by 20
  38. 2024Enhancing In-context Learning via Linear Probe Calibrationcited by 16

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 18.

25 papers.

Sort:

By area:

  1. 2025Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewardscited by 11
  2. 2025MultiSHAP: A Shapley-Based Framework for Explaining Cross-Modal Interactions in Multimodal AI Modelscited by 7
  3. 2025Theoretical Tensions in RLHF: Reconciling Empirical Success with Inconsistencies in Social Choice Theorycited by 6
  4. 2026Same Answer, Different Confidence: Protocol Sensitivity in LLM Confidence Calibrationcited by 6
  5. 2025Do Large Language Models (Really) Need Statistical Foundations?cited by 5
  6. 2026CATTO: Balancing Preferences and Confidence in Language Modelscited by 4
  7. 2026EvidenceRL: Reinforcing Evidence Consistency for Trustworthy Language Modelscited by 4
  8. 2026Calibration-Aware Policy Optimization for Reasoning LLMscited by 4
  9. 2025Honesty over Accuracy: Trustworthy Language Models through Reinforced Hesitationcited by 3
  10. 2026Just how sure are you? Improving Verbalized Uncertainty Calibration in Medical VQAcited by 3
  11. 2025SCOPE: Stochastic and Counterbiased Option Placement for Evaluating Large Language Modelscited by 2
  12. 2025IA2: Alignment with ICL Activations Improves Supervised Fine-Tuningcited by 1
13 more
  1. 2026Unlocking the Pre-Trained Model as a Dual-Alignment Calibrator for Post-Trained LLMscited by 1
  2. 2026Balancing Classification and Calibration Performance in Decision-Making LLMs via Calibration Aware Reinforcement Learningcited by 1
  3. 2026A Semantic-Sampling Framework for Evaluating Calibration in Open-Ended Question Answeringcited by 1
  4. 2026Reinforcement Learning over Predictive Distributions for LLM Regressioncited by 1
  5. 2026Confidence Calibration in Large Language Modelscited by 1
  6. 2026Beyond Logit Adjustment: A Residual Decomposition Framework for Long-Tailed Rerankingcited by 0
  7. 2026Discovery of Hidden Miscalibration Regimescited by 0
  8. 2026LaRA: Layer-wise Representation Analysis for Detecting Data Contamination in RL Post-Trainingcited by 0
  9. 2026Beyond Post-Hoc Temperature Scaling: Bilevel Optimization for LLM Calibrationcited by 0
  10. 2026Stable Miscalibration in Large Language Models: A Practical View of High-Confidence Errorscited by 0
  11. 2026Wrong Prediction, Right Answer: Recovering Evidence from Collapsed LLM Sequence Scorescited by 0
  12. 2026CHAIN: Calibrated LLM Forecasting via Causal-Temporal Hypergraph Inferencecited by 0
  13. 2026OmniMed-Jev: Calibrating LVLM Confidence for Trustworthy Medical Multimodal Decisions via System Onecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Does Alignment Tuning Really Break LLMs' Internal Confidence?
  2. 2023On the Calibration of Large Language Models and Alignment
  3. 2023On Diversified Preferences of Large Language Model Alignment
  4. 2023Investigating Uncertainty Calibration of Aligned Language Models under the Multiple-Choice Setting
  5. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment
  6. 2025Data Selection for LLM Alignment Using Fine-Grained Preferences
  7. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  8. 2024A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniques