# On the Calibration of Large Language Models and Alignment

arXiv: https://arxiv.org/abs/2311.13240 · HTML: https://mlsanity.com/paper/2311.13240/

## Key facts

- Cited by 73 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 50 references that are in the map.
- First posted on arXiv 22 Nov 2023.

## Builds on (most cited first, 30 of 35)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks](https://mlsanity.com/paper/1610.02136/) (2016)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Improving Factuality and Reasoning in Language Models through Multiagent Debate](https://mlsanity.com/paper/2305.14325/) (2023)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023)
- [Cross-Task Generalization via Natural Language Crowdsourcing Instructions](https://mlsanity.com/paper/2104.08773/) (2021)
- [Measuring Calibration in Deep Learning](https://mlsanity.com/paper/1904.01685/) (2019)
- [How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering](https://mlsanity.com/paper/2012.00955/) (2020)
- [Revisiting the Calibration of Modern Neural Networks](https://mlsanity.com/paper/2106.07998/) (2021)
- [Calibration of Pre-trained Transformers](https://mlsanity.com/paper/2003.07892/) (2020)
- [On Hallucination and Predictive Uncertainty in Conditional Language Generation](https://mlsanity.com/paper/2103.15025/) (2021)
- [Sources of Hallucination by Large Language Models on Inference Tasks](https://mlsanity.com/paper/2305.14552/) (2023)
- [ExpertPrompting: Instructing Large Language Models to be Distinguished Experts](https://mlsanity.com/paper/2305.14688/) (2023)
- [Training Trajectories of Language Models Across Scales](https://mlsanity.com/paper/2212.09803/) (2022)
- [Learning from Task Descriptions](https://mlsanity.com/paper/2011.08115/) (2020)
- [Posterior calibration and exploratory analysis for natural language processing models](https://mlsanity.com/paper/1508.05154/) (2015)

## Built on by (most cited first, 30 of 73)

- [A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directions](https://mlsanity.com/paper/2412.05563/) (2024)
- [Distillation Scaling Laws](https://mlsanity.com/paper/2502.08606/) (2025)
- [Are Reasoning Models More Prone to Hallucination?](https://mlsanity.com/paper/2505.23646/) (2025)
- [SimBench: Benchmarking the Ability of Large Language Models to Simulate Human Behaviors](https://mlsanity.com/paper/2510.17516/) (2025)
- [Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignment](https://mlsanity.com/paper/2412.14516/) (2024)
- [SeaKR: Self-aware Knowledge Retrieval for Adaptive Retrieval Augmented Generation](https://mlsanity.com/paper/2406.19215/) (2024)
- [Mind the Confidence Gap: Overconfidence, Calibration, and Distractor Effects in Large Language Models](https://mlsanity.com/paper/2502.11028/) (2025)
- [Thermometer: Towards Universal Calibration for Large Language Models](https://mlsanity.com/paper/2403.08819/) (2024)
- [LoRA Dropout as a Sparsity Regularizer for Overfitting Control](https://mlsanity.com/paper/2404.09610/) (2024)
- [On the Challenges and Opportunities in Generative AI](https://mlsanity.com/paper/2403.00025/) (2024)
- [On Diversified Preferences of Large Language Model Alignment](https://mlsanity.com/paper/2312.07401/) (2023)
- [Atomic Calibration of LLMs in Long-Form Generations](https://mlsanity.com/paper/2410.13246/) (2024)
- [A Survey on the Honesty of Large Language Models](https://mlsanity.com/paper/2409.18786/) (2024)
- [On Calibration of LLM-based Guard Models for Reliable Content Moderation](https://mlsanity.com/paper/2410.10414/) (2024)
- [Revisiting Uncertainty Estimation and Calibration of Large Language Models](https://mlsanity.com/paper/2505.23854/) (2025)
- [Aligning Large Language Models by On-Policy Self-Judgment](https://mlsanity.com/paper/2402.11253/) (2024)
- [Your Pre-trained LLM is Secretly an Unsupervised Confidence Calibrator](https://mlsanity.com/paper/2505.16690/) (2025)
- [Beyond Log Likelihood: Probability-Based Objectives for Supervised Fine-Tuning across the Model Capability Continuum](https://mlsanity.com/paper/2510.00526/) (2025)
- [LoVeC: Reinforcement Learning for Better Verbalized Confidence in Long-Form Generations](https://mlsanity.com/paper/2505.23912/) (2025)
- [Probabilities of Chat LLMs Are Miscalibrated but Still Predict Correctness on Multiple-Choice Q&A](https://mlsanity.com/paper/2402.13213/) (2024)
- [Contextualized Sequence Likelihood: Enhanced Confidence Scores for Natural Language Generation](https://mlsanity.com/paper/2406.01806/) (2024)
- [Sparse Logit Sampling: Accelerating Knowledge Distillation in LLMs](https://mlsanity.com/paper/2503.16870/) (2025)
- [Large Language Models Do Multi-Label Classification Differently](https://mlsanity.com/paper/2505.17510/) (2025)
- [Towards Objective Fine-tuning: How LLMs' Prior Knowledge Causes Potential Poor Calibration?](https://mlsanity.com/paper/2505.20903/) (2025)
- [Cascaded Language Models for Cost-effective Human-AI Decision-Making](https://mlsanity.com/paper/2506.11887/) (2025)
- [Scalable Bayesian Low-Rank Adaptation of Large Language Models via Stochastic Variational Subspace Inference](https://mlsanity.com/paper/2506.21408/) (2025)
- [The Confidence Paradox: Can LLM Know When It's Wrong](https://mlsanity.com/paper/2506.23464/) (2025)
- [Lexical Hints of Accuracy in LLM Reasoning Chains](https://mlsanity.com/paper/2508.15842/) (2025)
- [A Survey of Calibration Process for Black-Box LLMs](https://mlsanity.com/paper/2412.12767/) (2024)
- [BaseCal: Unsupervised Confidence Calibration via Base Model Signals](https://mlsanity.com/paper/2601.03042/) (2026)

## Abstract (arXiv)

As large language models attract increasing attention and find widespread application, concurrent challenges of reliability also arise at the same time. Confidence calibration, an effective analysis method for gauging the reliability of deep models, serves as a crucial tool for assessing and improving their reliability. However, such investigation has been comparatively underexplored. In this work, we conduct a systematic examination of the calibration of aligned language models throughout the entire construction process, including pretraining and alignment training. At each stage, we investigate how different training settings, such as parameter scales and training data, affect model calibration. To thoroughly assess model calibration, we evaluate models on three most concerned aspects: generation, factuality and understanding. Our work sheds light on whether popular LLMs are well-calibrated and how the training process influences model calibration.