# A Survey of Uncertainty Estimation Methods on Large Language Models

arXiv: https://arxiv.org/abs/2503.00172 · HTML: https://mlsanity.com/paper/2503.00172/

## Key facts

- Cited by 62 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 57 references that are in the map.
- First posted on arXiv 28 Feb 2025.

## Builds on (most cited first, 30 of 42)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [A Survey of Large Language Models](https://mlsanity.com/paper/2303.18223/) (2023)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [Crowdsourcing Multiple Choice Science Questions](https://mlsanity.com/paper/1707.06209/) (2017)
- [Teaching Models to Express Their Uncertainty in Words](https://mlsanity.com/paper/2205.14334/) (2022)
- [Large Language Models Can Self-Improve](https://mlsanity.com/paper/2210.11610/) (2022)
- [CodeBLEU: a Method for Automatic Evaluation of Code Synthesis](https://mlsanity.com/paper/2009.10297/) (2020)
- [How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answering](https://mlsanity.com/paper/2012.00955/) (2020)
- [Measuring short-form factuality in large language models](https://mlsanity.com/paper/2411.04368/) (2024)
- [INSIDE: LLMs' Internal States Retain the Power of Hallucination Detection](https://mlsanity.com/paper/2402.03744/) (2024)
- [A Survey of Confidence Estimation and Calibration in Large Language Models](https://mlsanity.com/paper/2311.08298/) (2023)
- [Unsupervised Quality Estimation for Neural Machine Translation](https://mlsanity.com/paper/2005.10608/) (2020)
- [Out-of-Distribution Detection and Selective Generation for Conditional Language Models](https://mlsanity.com/paper/2209.15558/) (2022)
- [Fact-Checking the Output of Large Language Models via Token-Level Uncertainty Quantification](https://mlsanity.com/paper/2403.04696/) (2024)
- [Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similarities](https://mlsanity.com/paper/2405.20003/) (2024)
- [Conformal Prediction with Large Language Models for Multi-Choice Question Answering](https://mlsanity.com/paper/2305.18404/) (2023)
- [LM-Polygraph: Uncertainty Estimation for Language Models](https://mlsanity.com/paper/2311.07383/) (2023)
- [Quantifying Uncertainty in Answers from any Language Model and Enhancing their Trustworthiness](https://mlsanity.com/paper/2308.16175/) (2023)
- [Decomposing Uncertainty for Large Language Models through Input Clarification Ensembling](https://mlsanity.com/paper/2311.08718/) (2023)
- [Enhancing Uncertainty-Based Hallucination Detection with Stronger Focus](https://mlsanity.com/paper/2311.13230/) (2023)

## Built on by (most cited first, 30 of 62)

- [MUR: Momentum Uncertainty guided Reasoning for Large Language Models](https://mlsanity.com/paper/2507.14958/) (2025)
- [MetaFaith: Faithful Natural Language Uncertainty Expression in LLMs](https://mlsanity.com/paper/2505.24858/) (2025)
- [Geometric Uncertainty for Detecting and Correcting Hallucinations in LLMs](https://mlsanity.com/paper/2509.13813/) (2025)
- [PiCSAR: Probabilistic Confidence Selection And Ranking for Reasoning Chains](https://mlsanity.com/paper/2508.21787/) (2025)
- [Complementing Self-Consistency with Cross-Model Disagreement for Uncertainty Quantification](https://mlsanity.com/paper/2604.17112/) (2026)
- [Every Response Counts: Quantifying Uncertainty of LLM-based Multi-Agent Systems through Tensor Decomposition](https://mlsanity.com/paper/2604.08708/) (2026)
- [NOVA: NOise-aware Verbal Confidence CAlibration for Robust Large Language Models in RAG Systems](https://mlsanity.com/paper/2601.11004/) (2026)
- [It's Not Always Sycophancy: Measuring LLM Conformity as a Function of Epistemic Uncertainty](https://mlsanity.com/paper/2605.27288/) (2026)
- [Verbalizing LLM's Higher-order Uncertainty via Imprecise Probabilities](https://mlsanity.com/paper/2603.10396/) (2026)
- [I-CALM: Incentivizing Confidence-Aware Abstention for LLM Selective Answering](https://mlsanity.com/paper/2604.03904/) (2026)
- [Comparing Uncertainty Measurement and Mitigation Methods for Large Language Models: A Systematic Review](https://mlsanity.com/paper/2504.18346/) (2025)
- [CritiCal: Can Critique Help LLM Uncertainty or Confidence Calibration?](https://mlsanity.com/paper/2510.24505/) (2025)
- [AI Alignment Breaks at the Edge](https://mlsanity.com/paper/2602.20042/) (2026)
- [Walking Through Uncertainty: An Empirical Study of Uncertainty Estimation for Audio-Aware Large Language Models](https://mlsanity.com/paper/2604.25591/) (2026)
- [Reinforcement Learning with Metacognitive Feedback Elicits Faithful Uncertainty Expression in LLMs](https://mlsanity.com/paper/2606.32032/) (2026)
- [STRUCTSURVEY: Structured Agentic Retrieval for Automated Survey Paper Generation](https://mlsanity.com/paper/2607.01243/) (2026)
- [Don't Throw Away Your Beams: Improving Consistency-based Uncertainties in LLMs via Beam Search](https://mlsanity.com/paper/2512.09538/) (2025)
- [Uncertainty Propagation in LLM-Based Systems](https://mlsanity.com/paper/2604.23505/) (2026)
- [Uncertainty Decomposition for Clarification Seeking in LLM Agents](https://mlsanity.com/paper/2606.19559/) (2026)
- [Toward Efficient Uncertainty in LLMs through Evidential Knowledge Distillation](https://mlsanity.com/paper/2507.18366/) (2025)
- [FinZero: Launching Multi-modal Financial Time Series Forecast with Large Reasoning Model](https://mlsanity.com/paper/2509.08742/) (2025)
- [LLM Reasoning Predicts When Models Are Right: Evidence from Coding Classroom Discourse](https://mlsanity.com/paper/2602.09832/) (2026)
- [Furina: Fragmented Uncertainty-Driven Refusal Instability Attack](https://mlsanity.com/paper/2605.26158/) (2026)
- [Can LLMs Use Linguistic Uncertainty Markers to Reliably Reflect Intrinsic Confidence?](https://mlsanity.com/paper/2605.28778/) (2026)
- [A Systematic Evaluation of Black-Box Uncertainty Estimation Methods for Large Language Models](https://mlsanity.com/paper/2606.19868/) (2026)
- [Aligning Language Models with Selective Prediction](https://mlsanity.com/paper/2607.03528/) (2026)
- [Last Step Matters: Early Uncertainty Cannot Predict Failure in Long-Horizon Agents](https://mlsanity.com/paper/2608.29685/) (2026)
- [MUCH: A Multilingual Claim Hallucination Benchmark](https://mlsanity.com/paper/2511.17081/) (2025)
- [Token-Level Marginalization for Multi-Label LLM Classifiers](https://mlsanity.com/paper/2511.22312/) (2025)
- [Efficient Epistemic Uncertainty Estimation for Large Language Models via Knowledge Distillation](https://mlsanity.com/paper/2602.01956/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have demonstrated remarkable capabilities across various tasks. However, these models could offer biased, hallucinated, or non-factual responses camouflaged by their fluency and realistic appearance. Uncertainty estimation is the key method to address this challenge. While research efforts in uncertainty estimation are ramping up, there is a lack of comprehensive and dedicated surveys on LLM uncertainty estimation. This survey presents four major avenues of LLM uncertainty estimation. Furthermore, we perform extensive experimental evaluations across multiple methods and datasets. At last, we provide critical and promising future directions for LLM uncertainty estimation.