mlsanityData to 7 Oct 2026

Does Alignment Tuning Really Break LLMs' Internal Confidence?

arXiv 2409.00352 ↗First posted 31 Aug 2024cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Large Language Models (LLMs) have shown remarkable progress, but their real-world application necessitates reliable calibration. This study conducts a comprehensive analysis of calibration degradation of LLMs across four dimensions: models, calibration metrics, tasks, and confidence extraction methods. Initial analysis showed that the relationship between alignment and calibration is not always a trade-off, but under stricter analysis conditions, we found the alignment process consistently harms calibration. This highlights the need for (1) a careful approach when measuring model confidences and calibration errors and (2) future research into algorithms that can help LLMs to achieve both instruction-following and calibration without sacrificing either.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 9 of its 15 references that are in the map. First posted on arXiv 31 Aug 2024.

Builds on

9 of 15 references.

References · 9

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  4. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  6. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  7. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  8. 2019Measuring Calibration in Deep Learningcited by 443
  9. 2023On the Calibration of Large Language Models and Alignmentcited by 72

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach
  2. 2024Calibrating Long-form Generations from Large Language Models
  3. 2025A Comprehensive Evaluation framework of Alignment Techniques for LLMs
  4. 2025Direct Confidence Alignment: Aligning Verbalized Confidence with Internal Confidence In Large Language Models
  5. 2023Investigating Uncertainty Calibration of Aligned Language Models under the Multiple-Choice Setting
  6. 2026Unlocking the Pre-Trained Model as a Dual-Alignment Calibrator for Post-Trained LLMs
  7. 2024Calibrating the Confidence of Large Language Models by Eliciting Fidelity
  8. 2026Exploring the Effects of Alignment on Numerical Bias in Large Language Models