mlsanityData to 7 Oct 2026

Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback

arXiv 2305.14975 ↗First posted 24 May 2023cs.CLcited by 788

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

A trustworthy real-world prediction system should produce well-calibrated confidence scores; that is, its confidence in an answer should be indicative of the likelihood that the answer is correct, enabling deferral to an expert in cases of low-confidence predictions. Recent studies have shown that unsupervised pre-training produces large language models (LMs) whose conditional probabilities are remarkably well-calibrated. However, the most widely-used LMs are fine-tuned with reinforcement learning from human feedback (RLHF-LMs), and some studies have suggested that RLHF-LMs produce conditional probabilities that are very poorly calibrated. In light of this perceived weakness, we conduct a broad evaluation of methods for extracting confidence scores from RLHF-LMs. For RLHF-LMs such as ChatGPT, GPT-4, and Claude, we find that verbalized confidences emitted as output tokens are typically better-calibrated than the model's conditional probabilities on the TriviaQA, SciQ, and TruthfulQA benchmarks, often reducing the expected calibration error by a relative 50%.

Key facts. Cited by 788 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 23 of its 30 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

23 of 30 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2017On Calibration of Modern Neural Networkscited by 4,860
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  8. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  9. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  10. 2020Learning to summarize from human feedbackcited by 2,562
  11. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  12. 2022Language Models (Mostly) Know What They Knowcited by 1,783
10 more
  1. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  2. 2017Selective Classification for Deep Neural Networkscited by 799
  3. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  4. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  5. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  6. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
  7. 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 209
  8. 2022Uncertainty Quantification with Pre-trained Language Models: A Large-Scale Empirical Analysiscited by 104
  9. 2021Representing Numbers in NLP: a Survey and a Visioncited by 83
  10. 2022On the Calibration of Pre-trained Language Models using Mixup Guided by Area Under the Margin and Saliencycited by 31

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 30; 2024: 175; 2025: 232; 2026: 351.

788 papers.

Sort:

By area:

  1. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  2. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  3. 2023ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMscited by 297
  4. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 253
  5. 2024LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinationscited by 230
  6. 2024Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?cited by 227
  7. 2023Fine-tuning Language Models for Factualitycited by 219
  8. 2024Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaborationcited by 176
  9. 2024Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similaritiescited by 156
  10. 2025Reasoning Models Know When They're Right: Probing Hidden States for Self-Verificationcited by 154
  11. 2024MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoningcited by 145
  12. 2024Large Legal Fictions: Profiling Legal Hallucinations in Large Language Modelscited by 144
38 more
  1. 2025AbstentionBench: Reasoning LLMs Fail on Unanswerable Questionscited by 129
  2. 2024Language Models with Conformal Factuality Guaranteescited by 128
  3. 2023Quantifying Uncertainty in Answers from any Language Model and Enhancing their Trustworthinesscited by 121
  4. 2024Empowering Biomedical Discovery with AI Agentscited by 117
  5. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
  6. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 115
  7. 2023Decomposing Uncertainty for Large Language Models through Input Clarification Ensemblingcited by 113
  8. 2023Bayesian Low-rank Adaptation for Large Language Modelscited by 112
  9. 2025Confidence Improves Self-Consistency in LLMscited by 111
  10. 2024What Large Language Models Know and What People Think They Knowcited by 109
  11. 2025Beyond Binary Rewards: Training LMs to Reason About Their Uncertaintycited by 107
  12. 2024Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertaintycited by 106
  13. 2024Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraphcited by 105
  14. 2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 105
  15. 2024Approaching Human-Level Forecasting with Language Modelscited by 102
  16. 2024SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationalescited by 102
  17. 2023Evaluation and Analysis of Hallucination in Large Vision-Language Modelscited by 99
  18. 2024Large Language Models Must Be Taught to Know What They Don't Knowcited by 99
  19. 2024Can LLM be a Personalized Judge?cited by 99
  20. 2024Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreementcited by 99
  21. 2025Maximizing Confidence Alone Improves Reasoningcited by 97
  22. 2024HaloScope: Harnessing Unlabeled LLM Generations for Hallucination Detectioncited by 96
  23. 2025Uncertainty Quantification and Confidence Calibration in Large Language Models: A Surveycited by 93
  24. 2023SAC3: Reliable Hallucination Detection in Black-Box Language Models via Semantic-aware Cross-check Consistencycited by 89
  25. 2024Unfamiliar Finetuning Examples Control How Language Models Hallucinatecited by 85
  26. 2024Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words?cited by 85
  27. 2024On Verbalized Confidence Scores for LLMscited by 83
  28. 2024Self-Alignment for Factuality: Mitigating Hallucinations in LLMs via Self-Evaluationcited by 79
  29. 2023Self-Evaluation Improves Selective Generation in Large Language Modelscited by 74
  30. 2025Efficient Test-Time Scaling via Self-Calibrationcited by 74
  31. 2024Uncertainty Estimation and Quantification for LLMs: A Simple Supervised Approachcited by 73
  32. 2023Label-free Node Classification on Graphs with Large Language Models (LLMS)cited by 72
  33. 2023On the Calibration of Large Language Models and Alignmentcited by 72
  34. 2023Alignment for Honestycited by 72
  35. 2024"I'm Not Sure, But...": Examining the Impact of Large Language Models' Uncertainty Expression on User Reliance and Trustcited by 67
  36. 2024Learning to Route LLMs with Confidence Tokenscited by 66
  37. 2024Rejection Improves Reliability: Training LLMs to Refuse Unknown Questions Using RL from Knowledge Feedbackcited by 64
  38. 2024Calibrating Large Language Models Using Their Generations Onlycited by 63

50 most cited of 788 All 788 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Test-time Calibration Learning for Large Language Model Reasoning
  2. 2026On the effectiveness of reward functions in reinforcement learning for confidence calibration of large language models
  3. 2026The ACUTE Protocol: Operationalizing Language Model Activations for Better Calibration, Utility, and Trust
  4. 2026CALIBER: Calibrating Confidence Before and After Reasoning in Language Models
  5. 2026Claim-Level Confidence Calibration for Reliable Decision Making with Large Language Models
  6. 2024Confidence Under the Hood: An Investigation into the Confidence-Probability Alignment in Large Language Models
  7. 2026Quantifying Faithful Confidence Expression in Large Reasoning Models
  8. 2025Direct Confidence Alignment: Aligning Verbalized Confidence with Internal Confidence In Large Language Models