mlsanityData to 14 Oct 2026

A Survey of Confidence Estimation and Calibration in Large Language Models

arXiv 2311.08298 ↗First posted 14 Nov 2023cs.CL cs.AIcited by 256

Not on the map. Why?

Cited by 34 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LLMs) have demonstrated remarkable capabilities across a wide range of tasks in various domains. Despite their impressive performance, they can be unreliable due to factual errors in their generations. Assessing their confidence and calibrating them across different tasks can help mitigate risks and enable LLMs to produce better generations. There has been a lot of recent research aiming to address this, but there has been no comprehensive overview to organize it and outline the main lessons learned. The present survey aims to bridge this gap. In particular, we outline the challenges and we summarize recent technical advancements for LLM confidence estimation and calibration. We further discuss their applications and suggest promising directions for future work.

Key facts. Cited by 256 papers in the map (data through 14 Oct 2026). Builds on 93 of its 138 references that are in the map. First posted on arXiv 14 Nov 2023.

Builds on

93 of 138 references.

References · 93

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  8. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  9. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  10. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  11. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  12. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
38 more
  1. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,475
  2. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  3. 2017On Calibration of Modern Neural Networkscited by 4,960
  4. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  5. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  6. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  11. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,352
  12. 2019Language Models as Knowledge Bases?cited by 1,936
  13. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  14. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,485
  15. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,434
  16. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  17. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  18. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  19. 2023PaLM 2 Technical Reportcited by 1,162
  20. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,102
  21. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  22. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  23. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 807
  24. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
  25. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 763
  26. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  27. 2023The Internal State of an LLM Knows When It's Lyingcited by 659
  28. 2020Recipes for building an open-domain chatbotcited by 593
  29. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
  30. 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 464
  31. 2019Measuring Calibration in Deep Learningcited by 446
  32. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 445
  33. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 401
  34. 2019Beyond temperature scaling: Obtaining well-calibrated multiclass probabilities with Dirichlet calibrationcited by 364
  35. 2020Calibrating Deep Neural Networks using Focal Losscited by 357
  36. 2020Calibration of Pre-trained Transformerscited by 315
  37. 2021Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detectioncited by 264
  38. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 263

50 most cited of 93 All 93 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 34; 2025: 97; 2026: 125.

256 papers.

Sort:

By area:

  1. 2025Deep Think with Confidencecited by 205
  2. 2025Scalable Best-of-N Selection for Large Language Models via Self-Certaintycited by 187
  3. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 121
  4. 2025Maximizing Confidence Alone Improves Reasoningcited by 100
  5. 2024What Did I Do Wrong? Quantifying LLMs' Sensitivity and Consistency to Prompt Engineeringcited by 90
  6. 2024Calibrating Large Language Models with Sample Consistencycited by 73
  7. 2025Reasoning Models Better Express Their Confidencecited by 65
  8. 2025Are Reasoning Models More Prone to Hallucination?cited by 64
  9. 2025A Survey of Uncertainty Estimation Methods on Large Language Modelscited by 62
  10. 2024A Survey of Uncertainty Estimation in LLMs: Theory Meets Practicecited by 55
  11. 2025Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of Large Language Modelscited by 52
  12. 2024Can Unconfident LLM Annotations Be Used for Confident Conclusions?cited by 49
38 more
  1. 2025Mind the Confidence Gap: Overconfidence, Calibration, and Distractor Effects in Large Language Modelscited by 40
  2. 2025LLM-as-a-Prophet: Understanding Predictive Intelligence with Prophet Arenacited by 40
  3. 2024FIRE: Fact-checking with Iterative Retrieval and Verificationcited by 39
  4. 2024A Stitch in Time Saves Nine: Small VLM is a Precise Guidance for Accelerating Large VLMscited by 35
  5. 2025Towards Trustworthy Retrieval Augmented Generation for Large Language Models: A Surveycited by 30
  6. 2024Cycles of Thought: Measuring LLM Confidence through Stable Explanationscited by 29
  7. 2024DriveGPT: Scaling Autoregressive Behavior Models for Drivingcited by 29
  8. 2024Confidence v.s. Critique: A Decomposition of Self-Correction Capability for LLMscited by 29
  9. 2024Calibrating Reasoning in Language Models with Internal Consistencycited by 28
  10. 2024Finetuning Language Models to Emit Linguistic Expressions of Uncertaintycited by 26
  11. 2024I Don't Know: Explicit Modeling of Uncertainty with an [IDK] Tokencited by 26
  12. 2025On Benchmarking Human-Like Intelligence in Machinescited by 25
  13. 2025Prototypical Human-AI Collaboration Behaviors from LLM-Assisted Writing in the Wildcited by 25
  14. 2025BroRL: Scaling Reinforcement Learning via Broadened Explorationcited by 25
  15. 2024A Survey on the Honesty of Large Language Modelscited by 23
  16. 2025Revisiting Uncertainty Estimation and Calibration of Large Language Modelscited by 23
  17. 2025A Head to Predict and a Head to Question: Pre-trained Uncertainty Quantification Heads for Hallucination Detection in LLM Outputscited by 21
  18. 2025Reliable Text-to-SQL with Adaptive Abstentioncited by 20
  19. 2025An Empirical Analysis of Uncertainty in Large Language Model Evaluationscited by 20
  20. 2025Token-Level Density-Based Uncertainty Quantification Methods for Eliciting Truthfulness of Large Language Modelscited by 18
  21. 2026Intelligent AI Delegationcited by 18
  22. 2024METIS: Fast Quality-Aware RAG Systems with Configuration Adaptationcited by 17
  23. 2024Probabilities of Chat LLMs Are Miscalibrated but Still Predict Correctness on Multiple-Choice Q&Acited by 16
  24. 2024Knowledge Entropy Decay during Language Model Pretraining Hinders New Knowledge Acquisitioncited by 16
  25. 2025Efficient Hallucination Detection for LLMs Using Uncertainty-Aware Attention Headscited by 16
  26. 2025Revisiting Epistemic Markers in Confidence Estimation: Can Markers Accurately Reflect Large Language Models' Uncertainty?cited by 16
  27. 2025MetaFaith: Faithful Natural Language Uncertainty Expression in LLMscited by 16
  28. 2025Utility-Diversity Aware Online Batch Selection for LLM Supervised Fine-tuningcited by 15
  29. 2026Agentic Confidence Calibrationcited by 15
  30. 2025Do LLMs Really Forget? Evaluating Unlearning with Knowledge Correlation and Confidence Awarenesscited by 14
  31. 2025Query-Level Uncertainty in Large Language Modelscited by 14
  32. 2025Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missingcited by 14
  33. 2025Calibrating Verbalized Confidence with Self-Generated Distractorscited by 14
  34. 2025When Models Lie, We Learn: Multilingual Span-Level Hallucination Detection with PsiloQAcited by 14
  35. 2025Can LLMs Predict Their Own Failures? Self-Awareness via Internal Circuitscited by 14
  36. 2026LLMs Encode Their Failures: Predicting Success from Pre-Generation Activationscited by 14
  37. 2026Calibrated Decisions at Scale: Converting Police Crash Narratives into Probabilistic Crash Variables with a System One Model (Jev)cited by 14
  38. 2024Self-Consistency Boosts Calibration for Math Reasoningcited by 13

50 most cited of 256 All 256 (JSON)