# Calibrating the Confidence of Large Language Models by Eliciting Fidelity

arXiv: https://arxiv.org/abs/2404.02655 · HTML: https://mlsanity.com/paper/2404.02655/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 20 of its 36 references that are in the map.
- First posted on arXiv 3 Apr 2024.

## Placed in

- LLM alignment / RLHF / preference optimisation / [LLM preference optimization methods](https://mlsanity.com/alignment/llm-preference-optimization-methods/)

## Builds on (most cited first, 20 of 20)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [A General Language Assistant as a Laboratory for Alignment](https://mlsanity.com/paper/2112.00861/) (2021)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023)
- [Teaching Models to Express Their Uncertainty in Words](https://mlsanity.com/paper/2205.14334/) (2022)
- [How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering](https://mlsanity.com/paper/2012.00955/) (2020)
- [Do Large Language Models Know What They Don't Know?](https://mlsanity.com/paper/2305.18153/) (2023)
- [Conformal Prediction with Large Language Models for Multi-Choice Question Answering](https://mlsanity.com/paper/2305.18404/) (2023)
- [Investigating Uncertainty Calibration of Aligned Language Models under the Multiple-Choice Setting](https://mlsanity.com/paper/2310.11732/) (2023)
- [The Open-World Lottery Ticket Hypothesis for OOD Intent Classification](https://mlsanity.com/paper/2210.07071/) (2022)

## Built on by (most cited first, 30 of 47)

- [Knowledge Boundary of Large Language Models: A Survey](https://mlsanity.com/paper/2412.12472/) (2024)
- [Generalization Bias in Large Language Model Summarization of Scientific Research](https://mlsanity.com/paper/2504.00025/) (2025)
- [Analyzing Uncertainty of LLM-as-a-Judge: Interval Evaluations with Conformal Prediction](https://mlsanity.com/paper/2509.18658/) (2025)
- [Counterfactual Debating with Preset Stances for Hallucination Elimination of LLMs](https://mlsanity.com/paper/2406.11514/) (2024)
- [Query-Level Uncertainty in Large Language Models](https://mlsanity.com/paper/2506.09669/) (2025)
- [Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missing](https://mlsanity.com/paper/2507.10587/) (2025)
- [Calibrating Verbalized Confidence with Self-Generated Distractors](https://mlsanity.com/paper/2509.25532/) (2025)
- [Read Your Own Mind: Reasoning Helps Surface Self-Confidence Signals in LLMs](https://mlsanity.com/paper/2505.23845/) (2025)
- [Beyond the Final Layer: Intermediate Representations for Better Multilingual Calibration in Large Language Models](https://mlsanity.com/paper/2510.03136/) (2025)
- [1+1>2: Can Large Language Models Serve as Cross-Lingual Knowledge Aggregators?](https://mlsanity.com/paper/2406.14721/) (2024)
- [Percept-WAM: Perception-Enhanced World-Awareness-Action Model for Robust End-to-End Autonomous Driving](https://mlsanity.com/paper/2511.19221/) (2025)
- [BaseCal: Unsupervised Confidence Calibration via Base Model Signals](https://mlsanity.com/paper/2601.03042/) (2026)
- [Are LLM Decisions Faithful to Verbal Confidence?](https://mlsanity.com/paper/2601.07767/) (2026)
- [Loki's Dance of Illusions: A Comprehensive Survey of Hallucination in Large Language Models](https://mlsanity.com/paper/2507.02870/) (2025)
- [Conformal Information Pursuit for Interactively Guiding Large Language Models](https://mlsanity.com/paper/2507.03279/) (2025)
- [DAWN-ICL: Strategic Planning of Problem-solving Trajectories for Zero-Shot In-Context Learning](https://mlsanity.com/paper/2410.20215/) (2024)
- [Mind the Generation Process: Fine-Grained Confidence Estimation During LLM Generation](https://mlsanity.com/paper/2508.12040/) (2025)
- [Robust Sampling for Active Statistical Inference](https://mlsanity.com/paper/2511.08991/) (2025)
- [LLM Abstention Can Be a Prompt Artifact, in Addition to Genuine Uncertainty](https://mlsanity.com/paper/2507.16199/) (2025)
- [BrowseConf: Confidence-Guided Test-Time Scaling for Web Agents](https://mlsanity.com/paper/2510.23458/) (2025)
- [Showing LLM-Generated Code Selectively Based on Confidence of LLMs](https://mlsanity.com/paper/2410.03234/) (2024)
- [Comparing Uncertainty Measurement and Mitigation Methods for Large Language Models: A Systematic Review](https://mlsanity.com/paper/2504.18346/) (2025)
- [Interaction Protocol Shapes Moral Judgment in Multi-Agent Debate](https://mlsanity.com/paper/2510.10002/) (2025)
- [Calibration Is Not Enough: Evaluating Confidence Estimation Under Language Variations](https://mlsanity.com/paper/2601.08064/) (2026)
- [Benchmarking Large Language Model Uncertainty for Prompt Optimization](https://mlsanity.com/paper/2409.10044/) (2024)
- [TuringQ: Benchmarking AI Comprehension in Theory of Computation](https://mlsanity.com/paper/2410.06547/) (2024)
- [Enhancing Reasoning Skills in Small Persian Medical Language Models Can Outperform Large-Scale Data Training](https://mlsanity.com/paper/2510.20059/) (2025)
- [Recursive Think-Answer Process for LLMs and VLMs](https://mlsanity.com/paper/2603.02099/) (2026)
- [A Systematic Evaluation of Black-Box Uncertainty Estimation Methods for Large Language Models](https://mlsanity.com/paper/2606.19868/) (2026)
- [Aligning Language Models with Selective Prediction](https://mlsanity.com/paper/2607.03528/) (2026)

## Abstract (arXiv)

Large language models optimized with techniques like RLHF have achieved good alignment in being helpful and harmless. However, post-alignment, these language models often exhibit overconfidence, where the expressed confidence does not accurately calibrate with their correctness rate. In this paper, we decompose the language model confidence into the textitUncertainty about the question and the textitFidelity to the answer generated by language models. Then, we propose a plug-and-play method to estimate the confidence of language models. Our method has shown good calibration performance by conducting experiments with 6 RLHF-LMs on four MCQA datasets. Moreover, we propose two novel metrics, IPR and CE, to evaluate the calibration of the model, and we have conducted a detailed discussion on textitTruly Well-Calibrated Confidence. Our method could serve as a strong baseline, and we hope that this work will provide some insights into the model confidence calibration.