mlsanityData to 7 Oct 2026

Calibrating the Confidence of Large Language Models by Eliciting Fidelity

arXiv 2404.02655 ↗First posted 3 Apr 2024cs.CLcited by 47

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Large language models optimized with techniques like RLHF have achieved good alignment in being helpful and harmless. However, post-alignment, these language models often exhibit overconfidence, where the expressed confidence does not accurately calibrate with their correctness rate. In this paper, we decompose the language model confidence into the textitUncertainty about the question and the textitFidelity to the answer generated by language models. Then, we propose a plug-and-play method to estimate the confidence of language models. Our method has shown good calibration performance by conducting experiments with 6 RLHF-LMs on four MCQA datasets. Moreover, we propose two novel metrics, IPR and CE, to evaluate the calibration of the model, and we have conducted a detailed discussion on textitTruly Well-Calibrated Confidence. Our method could serve as a strong baseline, and we hope that this work will provide some insights into the model confidence calibration.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 20 of its 36 references that are in the map. First posted on arXiv 3 Apr 2024.

Builds on

20 of 36 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Investigating Uncertainty Calibration of Aligned Language Models under the Multiple-Choice Setting18 Oct
  2. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback24 May
  3. 2022Training language models to follow instructions with human feedback4 Mar
  4. 2021A General Language Assistant as a Laboratory for Alignment1 Dec

Other references · 16

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2017On Calibration of Modern Neural Networkscited by 4,860
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  8. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  9. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  10. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  11. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  12. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
4 more
  1. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  2. 2023Do Large Language Models Know What They Don't Know?cited by 238
  3. 2023Conformal Prediction with Large Language Models for Multi-Choice Question Answeringcited by 131
  4. 2022The Open-World Lottery Ticket Hypothesis for OOD Intent Classificationcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 20; 2026: 20.

47 papers.

Sort:

By area:

  1. 2024Knowledge Boundary of Large Language Models: A Surveycited by 36
  2. 2025Generalization Bias in Large Language Model Summarization of Scientific Researchcited by 30
  3. 2025Analyzing Uncertainty of LLM-as-a-Judge: Interval Evaluations with Conformal Predictioncited by 26
  4. 2024Counterfactual Debating with Preset Stances for Hallucination Elimination of LLMscited by 17
  5. 2025Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missingcited by 14
  6. 2025Calibrating Verbalized Confidence with Self-Generated Distractorscited by 14
  7. 2025Query-Level Uncertainty in Large Language Modelscited by 12
  8. 2025Read Your Own Mind: Reasoning Helps Surface Self-Confidence Signals in LLMscited by 10
  9. 20241+1>2: Can Large Language Models Serve as Cross-Lingual Knowledge Aggregators?cited by 9
  10. 2025Beyond the Final Layer: Intermediate Representations for Better Multilingual Calibration in Large Language Modelscited by 9
  11. 2025Percept-WAM: Perception-Enhanced World-Awareness-Action Model for Robust End-to-End Autonomous Drivingcited by 9
  12. 2025Conformal Information Pursuit for Interactively Guiding Large Language Modelscited by 7
35 more
  1. 2026BaseCal: Unsupervised Confidence Calibration via Base Model Signalscited by 7
  2. 2024DAWN-ICL: Strategic Planning of Problem-solving Trajectories for Zero-Shot In-Context Learningcited by 6
  3. 2025Loki's Dance of Illusions: A Comprehensive Survey of Hallucination in Large Language Modelscited by 6
  4. 2026Are LLM Decisions Faithful to Verbal Confidence?cited by 6
  5. 2025Mind the Generation Process: Fine-Grained Confidence Estimation During LLM Generationcited by 5
  6. 2025BrowseConf: Confidence-Guided Test-Time Scaling for Web Agentscited by 5
  7. 2025Robust Sampling for Active Statistical Inferencecited by 5
  8. 2024Showing LLM-Generated Code Selectively Based on Confidence of LLMscited by 4
  9. 2025LLM Abstention Can Be a Prompt Artifact, in Addition to Genuine Uncertaintycited by 4
  10. 2025Comparing Uncertainty Measurement and Mitigation Methods for Large Language Models: A Systematic Reviewcited by 3
  11. 2026Calibration Is Not Enough: Evaluating Confidence Estimation Under Language Variationscited by 3
  12. 2024Benchmarking Large Language Model Uncertainty for Prompt Optimizationcited by 2
  13. 2024TuringQ: Benchmarking AI Comprehension in Theory of Computationcited by 2
  14. 2025Interaction Protocol Shapes Moral Judgment in Multi-Agent Debatecited by 2
  15. 2026Recursive Think-Answer Process for LLMs and VLMscited by 2
  16. 2026A Systematic Evaluation of Black-Box Uncertainty Estimation Methods for Large Language Modelscited by 2
  17. 2025Express Your Doubts -- Probabilistic World Modeling Should not be Based on Token logprobscited by 1
  18. 2025Quantifying LLM Biases Across Instruction Boundary in Mixed Question Formscited by 1
  19. 2025Enhancing Reasoning Skills in Small Persian Medical Language Models Can Outperform Large-Scale Data Trainingcited by 1
  20. 2026MarketBench: Evaluating AI Agents as Market Participantscited by 1
  21. 2026Ranking-Aware Calibration for Reliable Multimodal Reinforcement Learningcited by 1
  22. 2026Speaking in Self-Assessing Tongues: On the Verbalized Confidence of LLMs in Machine Translationcited by 1
  23. 2026Aligning Language Models with Selective Predictioncited by 1
  24. 2025Uncertainty-Aware Calibrated Clinical Text Classification with Large Language Modelscited by 0
  25. 2026Deep Learning-based Method for Expressing Knowledge Boundary of Black-Box LLMcited by 0
  26. 2026Beyond Confidence: The Rhythms of Reasoning in Generative Modelscited by 0
  27. 2026Assessing and Mitigating Miscalibration in LLM-Based Social Science Measurementcited by 0
  28. 2026PinPoint: Prompting with Informative Interior Pointscited by 0
  29. 2026Auto-Fill: Learning to Predict Missing Values Accurately with Specialist Language Modelscited by 0
  30. 2026Evaluation Pitfalls and Sparsity Limitations in LLM-based Confidence Estimates for Classificationcited by 0
  31. 2026Is This Your Final Answer? Cross-Contextual Consistency as a Measure of LLM Credibilitycited by 0
  32. 2026Are You Sure You're Sure? On the Impact of Instruction Tuning on Confidence and Lexical Diversitycited by 0
  33. 2026The Asymmetric Harms of LLM Compressioncited by 0
  34. 2026DEEPO: Dual-Entropy Enhanced Policy Optimization for Hallucination in MLLMscited by 0
  35. 2026CHAIN: Calibrated LLM Forecasting via Causal-Temporal Hypergraph Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ConfidenceBench: Evaluating Confidence Calibration in Large Language Models
  2. 2024Enhancing Language Model Factuality via Activation-Based Confidence Calibration and Guided Decoding
  3. 2023On the Calibration of Large Language Models and Alignment
  4. 2024Calibrating Long-form Generations from Large Language Models
  5. 2025Mind the Confidence Gap: Overconfidence, Calibration, and Distractor Effects in Large Language Models
  6. 2025Direct Confidence Alignment: Aligning Verbalized Confidence with Internal Confidence In Large Language Models
  7. 2025Trusted Uncertainty in Large Language Models: A Unified Framework for Confidence Calibration and Risk-Controlled Refusal
  8. 2024Graph-based Confidence Calibration for Large Language Models