mlsanityData to 7 Oct 2026

On the Calibration of Large Language Models and Alignment

arXiv 2311.13240 ↗First posted 22 Nov 2023cs.CLcited by 72

Not on the map. Why?

Abstract (arXiv)

As large language models attract increasing attention and find widespread application, concurrent challenges of reliability also arise at the same time. Confidence calibration, an effective analysis method for gauging the reliability of deep models, serves as a crucial tool for assessing and improving their reliability. However, such investigation has been comparatively underexplored. In this work, we conduct a systematic examination of the calibration of aligned language models throughout the entire construction process, including pretraining and alignment training. At each stage, we investigate how different training settings, such as parameter scales and training data, affect model calibration. To thoroughly assess model calibration, we evaluate models on three most concerned aspects: generation, factuality and understanding. Our work sheds light on whether popular LLMs are well-calibrated and how the training process influences model calibration.

Key facts. Cited by 72 papers in the map (data through 7 Oct 2026). Builds on 35 of its 50 references that are in the map. First posted on arXiv 22 Nov 2023.

Builds on

35 of 50 references.

References · 35

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2017On Calibration of Modern Neural Networkscited by 4,860
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  11. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  12. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
23 more
  1. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,362
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  3. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  4. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  5. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  6. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 973
  7. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  8. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  9. 2019Measuring Calibration in Deep Learningcited by 443
  10. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  11. 2021Revisiting the Calibration of Modern Neural Networkscited by 387
  12. 2020Calibration of Pre-trained Transformerscited by 307
  13. 2021On Hallucination and Predictive Uncertainty in Conditional Language Generationcited by 176
  14. 2023Sources of Hallucination by Large Language Models on Inference Taskscited by 168
  15. 2023ExpertPrompting: Instructing Large Language Models to be Distinguished Expertscited by 140
  16. 2022Training Trajectories of Language Models Across Scalescited by 80
  17. 2020Learning from Task Descriptionscited by 78
  18. 2015Posterior calibration and exploratory analysis for natural language processing modelscited by 75
  19. 2019Calibration of Encoder Decoder Models for Neural Machine Translationcited by 67
  20. 2019Sticking to the Facts: Confident Decoding for Faithful Data-to-Text Generationcited by 62
  21. 2023Why Does ChatGPT Fall Short in Providing Truthful Answers?cited by 54
  22. 2023kNN Prompting: Beyond-Context Learning with Calibration-Free Nearest Neighbor Inferencecited by 47
  23. 2023Preserving Pre-trained Features Helps Calibrate Fine-tuned Language Modelscited by 29

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 17; 2025: 30; 2026: 24.

72 papers.

Sort:

By area:

  1. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 115
  2. 2025Are Reasoning Models More Prone to Hallucination?cited by 62
  3. 2025Distillation Scaling Lawscited by 59
  4. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignmentcited by 50
  5. 2025SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviorscited by 48
  6. 2024SeaKR: Self-aware Knowledge Retrieval for Adaptive Retrieval Augmented Generationcited by 40
  7. 2025Mind the Confidence Gap: Overconfidence, Calibration, and Distractor Effects in Large Language Modelscited by 39
  8. 2024Thermometer: Towards Universal Calibration for Large Language Modelscited by 38
  9. 2024LoRA Dropout as a Sparsity Regularizer for Overfitting Controlcited by 35
  10. 2024On the Challenges and Opportunities in Generative AIcited by 31
  11. 2023On Diversified Preferences of Large Language Model Alignmentcited by 26
  12. 2024Atomic Calibration of LLMs in Long-Form Generationscited by 23
38 more
  1. 2025Revisiting Uncertainty Estimation and Calibration of Large Language Modelscited by 22
  2. 2024A Survey on the Honesty of Large Language Modelscited by 21
  3. 2024On Calibration of LLM-based Guard Models for Reliable Content Moderationcited by 21
  4. 2024Aligning Large Language Models by On-Policy Self-Judgmentcited by 20
  5. 2025Your Pre-trained LLM is Secretly an Unsupervised Confidence Calibratorcited by 19
  6. 2025LoVeC: Reinforcement Learning for Better Verbalized Confidence in Long-Form Generationscited by 17
  7. 2024Probabilities of Chat LLMs Are Miscalibrated but Still Predict Correctness on Multiple-Choice Q&Acited by 16
  8. 2024Contextualized Sequence Likelihood: Enhanced Confidence Scores for Natural Language Generationcited by 16
  9. 2025Beyond Log Likelihood: Probability-Based Objectives for Supervised Fine-Tuning across the Model Capability Continuumcited by 16
  10. 2025Sparse Logit Sampling: Accelerating Knowledge Distillation in LLMscited by 14
  11. 2025Large Language Models Do Multi-Label Classification Differentlycited by 12
  12. 2025Towards Objective Fine-tuning: How LLMs' Prior Knowledge Causes Potential Poor Calibration?cited by 11
  13. 2025Cascaded Language Models for Cost-effective Human-AI Decision-Makingcited by 11
  14. 2025Scalable Bayesian Low-Rank Adaptation of Large Language Models via Stochastic Variational Subspace Inferencecited by 11
  15. 2025The Confidence Paradox: Can LLM Know When It's Wrongcited by 11
  16. 2024A Survey of Calibration Process for Black-Box LLMscited by 8
  17. 2025Lexical Hints of Accuracy in LLM Reasoning Chainscited by 8
  18. 2024Transferable and Efficient Non-Factual Content Detection via Probe Training with Offline Consistency Checkingcited by 7
  19. 2025Advancing the Scientific Method with Large Language Models: From Hypothesis to Discoverycited by 7
  20. 2025Conformal Information Pursuit for Interactively Guiding Large Language Modelscited by 7
  21. 2026BaseCal: Unsupervised Confidence Calibration via Base Model Signalscited by 7
  22. 2024Diver: Large Language Model Decoding with Span-Level Mutual Information Verificationcited by 5
  23. 2025None of the Above, Less of the Right: Parallel Patterns between Humans and LLMs on Multi-Choice Questions Answeringcited by 5
  24. 2026Large Language Models Are Overconfident in Their Own Responsescited by 5
  25. 2025Logits are All We Need to Adapt Closed Modelscited by 4
  26. 2025VAQUUM: Are Vague Quantifiers Grounded in Visual Data?cited by 4
  27. 2025Learning from Reference Answers: Versatile Language Model Alignment without Binary Human Preference Datacited by 4
  28. 2025RankAlign: A Ranking View of the Generator-Validator Gap in Large Language Modelscited by 4
  29. 2026CATTO: Balancing Preferences and Confidence in Language Modelscited by 4
  30. 2026Alignment Makes Language Models Normative, Not Descriptivecited by 4
  31. 2026Walking Through Uncertainty: An Empirical Study of Uncertainty Estimation for Audio-Aware Large Language Modelscited by 4
  32. 2024Uncertainty Quantification on Graph Learning: A Surveycited by 3
  33. 2025Comparing Uncertainty Measurement and Mitigation Methods for Large Language Models: A Systematic Reviewcited by 3
  34. 2026Calibration Is Not Enough: Evaluating Confidence Estimation Under Language Variationscited by 3
  35. 2025The Capabilities and Limitations of Weak-to-Strong Generalization: Generalization and Calibrationcited by 2
  36. 2025Feeding LLM Annotations to BERT Classifiers at Your Own Riskcited by 2
  37. 2025Bi-directional Model Cascading with Proxy Confidencecited by 2
  38. 2025Bounds of Chain-of-Thought Robustness: Reasoning Steps, Embed Norms, and Beyondcited by 2

50 most cited of 72 All 72 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Calibrating Long-form Generations from Large Language Models
  2. 2025Restoring Calibration for Aligned Large Language Models: A Calibration-Aware Fine-Tuning Approach
  3. 2024Calibrating the Confidence of Large Language Models by Eliciting Fidelity
  4. 2023Investigating Uncertainty Calibration of Aligned Language Models under the Multiple-Choice Setting
  5. 2025Direct Confidence Alignment: Aligning Verbalized Confidence with Internal Confidence In Large Language Models
  6. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges
  7. 2025Human-Alignment and Calibration of Inference-Time Uncertainty in Large Language Models
  8. 2026Unlocking the Pre-Trained Model as a Dual-Alignment Calibrator for Post-Trained LLMs