mlsanityData to 7 Oct 2026

Investigating Uncertainty Calibration of Aligned Language Models under the Multiple-Choice Setting

arXiv 2310.11732 ↗First posted 18 Oct 2023cs.LG cs.CLcited by 20

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Despite the significant progress made in practical applications of aligned language models (LMs), they tend to be overconfident in output answers compared to the corresponding pre-trained LMs. In this work, we systematically evaluate the impact of the alignment process on logit-based uncertainty calibration of LMs under the multiple-choice setting. We first conduct a thoughtful empirical study on how aligned LMs differ in calibration from their pre-trained counterparts. Experimental results reveal that there are two distinct uncertainties in LMs under the multiple-choice setting, which are responsible for the answer decision and the format preference of the LMs, respectively. Then, we investigate the role of these two uncertainties on aligned LM's calibration through fine-tuning in simple synthetic alignment schemes and conclude that one reason for aligned LMs' overconfidence is the conflation of these two types of uncertainty. Furthermore, we examine the utility of common post-hoc calibration methods for aligned LMs and propose an easy-to-implement and sample-efficient method to calibrate aligned LMs. We hope our findings could provide insights into the design of more reliable alignment processes for LMs.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 30 of its 45 references that are in the map. First posted on arXiv 18 Oct 2023.

Builds on

30 of 45 references.

Closest ancestors · 3

Same direction, newest first

  1. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback24 May
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructions20 Dec
  3. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 27

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2017On Calibration of Modern Neural Networkscited by 4,860
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  10. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,335
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  12. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
15 more
  1. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  2. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  3. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  4. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  5. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  6. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  7. 2018Predictive Uncertainty Estimation via Prior Networkscited by 555
  8. 2019Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classificationcited by 396
  9. 2023Larger language models do in-context learning differentlycited by 386
  10. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 288
  11. 2023Generating with Confidence: Uncertainty Quantification for Black-box Large Language Modelscited by 286
  12. 2020Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learningcited by 216
  13. 2020Mix-n-Match: Ensemble and Compositional Methods for Uncertainty Calibration in Deep Learningcited by 190
  14. 2023Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchillacited by 145
  15. 2023DeepSpeed-Chat: Easy, Fast and Affordable RLHF Training of ChatGPT-like Models at All Scalescited by 107

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 11; 2025: 3; 2026: 6.

20 papers.

Sort:

By area:

  1. 2024Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaborationcited by 176
  2. 2024Uncertainty-Based Abstention in LLMs Improves Safety and Reduces Hallucinationscited by 51
  3. 2025Specializing Large Language Models to Simulate Survey Response Distributions for Global Populationscited by 50
  4. 2024Calibrating the Confidence of Large Language Models by Eliciting Fidelitycited by 47
  5. 2024Unveiling Uncertainty: A Deep Dive into Calibration and Performance of Multimodal Large Language Modelscited by 17
  6. 2024Probabilities of Chat LLMs Are Miscalibrated but Still Predict Correctness on Multiple-Choice Q&Acited by 16
  7. 2024Contextualized Sequence Likelihood: Enhanced Confidence Scores for Natural Language Generationcited by 16
  8. 2026Intelligent AI Delegationcited by 16
  9. 2024Teaching LLMs to Abstain across Languages via Multilingual Feedbackcited by 14
  10. 2024Listen Again and Choose the Right Answer: A New Paradigm for Automatic Speech Recognition with Large Language Modelscited by 13
  11. 2024Uncertainty is Fragile: Manipulating Uncertainty in Large Language Modelscited by 13
  12. 2024Unified Active Retrieval for Retrieval Augmented Generationcited by 8
8 more
  1. 2025AmbiK: Dataset of Ambiguous Tasks in Kitchen Environmentcited by 8
  2. 2024Enhancing Healthcare LLM Trust with Atypical Presentations Recalibrationcited by 6
  3. 2026BAS: A Decision-Theoretic Approach to Evaluating Large Language Model Confidencecited by 5
  4. 2026Wired for Overconfidence: A Mechanistic Perspective on Inflated Verbalized Confidence in LLMscited by 4
  5. 2025A Survey of LLM Alignment: Instruction Understanding, Intention Reasoning, and Reliable Generationcited by 3
  6. 2026Don't Blink: Evidence Collapse during Multimodal Reasoningcited by 2
  7. 2026Beyond Post-Hoc Temperature Scaling: Bilevel Optimization for LLM Calibrationcited by 0
  8. 2026Jailbreaks for Black-Box Uncertainty Quantification in Large Reasoning Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023On the Calibration of Large Language Models and Alignment
  2. 2025Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach
  3. 2024Does Alignment Tuning Really Break LLMs' Internal Confidence?
  4. 2025Human-Alignment and Calibration of Inference-Time Uncertainty in Large Language Models
  5. 2026The Alignment Tax: Response Homogenization in Aligned LLMs and Its Implications for Uncertainty Estimation
  6. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  7. 2024Aligner: Efficient Alignment by Learning to Correct
  8. 2025Direct Confidence Alignment: Aligning Verbalized Confidence with Internal Confidence In Large Language Models