mlsanityData to 14 Oct 2026

Systematic Evaluation of Uncertainty Estimation Methods in Large Language Models

arXiv 2510.20460 ↗First posted 23 Oct 2025cs.CL stat.AP stat.MEcited by 3

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) produce outputs with varying levels of uncertainty, and, just as often, varying levels of correctness; making their practical reliability far from guaranteed. To quantify this uncertainty, we systematically evaluate four approaches for confidence estimation in LLM outputs: VCE, MSP, Sample Consistency, and CoCoA (Vashurin et al., 2025). For the evaluation of the approaches, we conduct experiments on four question-answering tasks using a state-of-the-art open-source LLM. Our results show that each uncertainty metric captures a different facet of model confidence and that the hybrid CoCoA approach yields the best reliability overall, improving both calibration and discrimination of correct answers. We discuss the trade-offs of each method and provide recommendations for selecting uncertainty measures in LLM applications.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 22 of its 32 references that are in the map. First posted on arXiv 23 Oct 2025.

Builds on

22 of 32 references.

References · 22

  1. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  4. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  5. 2017On Calibration of Modern Neural Networkscited by 4,960
  6. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  7. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  9. 2016A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networkscited by 2,385
  10. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  11. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  12. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
10 more
  1. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  2. 2017Selective Classification for Deep Neural Networkscited by 852
  3. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 445
  4. 2020Calibration of Pre-trained Transformerscited by 315
  5. 2019Verified Uncertainty Calibrationcited by 309
  6. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 256
  7. 2018To Trust Or Not To Trust A Classifiercited by 207
  8. 2018Analyzing Uncertainty in Neural Machine Translationcited by 126
  9. 2024Calibrating Large Language Models with Sample Consistencycited by 73
  10. 2025Uncertainty Quantification for LLMs through Minimum Bayes Risk: Bridging Confidence and Consistencycited by 30

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Guideline-Grounded Evidence Accumulation for High-Stakes Agent Verificationcited by 5
  2. 2026I-CALM: Incentivizing Confidence-Aware Abstention for LLM Selective Answeringcited by 5
  3. 2026When Do We Need LLMs? A Diagnostic for Language-Driven Banditscited by 0