mlsanityData to 7 Oct 2026

To Believe or Not to Believe Your LLM

arXiv 2406.02543 ↗First posted 4 Jun 2024cs.LG cs.AI cs.CLcited by 120

Not on the map. Why?

Cited by 21 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We explore uncertainty quantification in large language models (LLMs), with the goal to identify when uncertainty in responses given a query is large. We simultaneously consider both epistemic and aleatoric uncertainties, where the former comes from the lack of knowledge about the ground truth (such as about facts or the language), and the latter comes from irreducible randomness (such as multiple possible answers). In particular, we derive an information-theoretic metric that allows to reliably detect when only epistemic uncertainty is large, in which case the output of the model is unreliable. This condition can be computed based solely on the output of the model obtained simply by some special iterative prompting based on the previous responses. Such quantification, for instance, allows to detect hallucinations (cases when epistemic uncertainty is high) in both single- and multi-answer responses. This is in contrast to many standard uncertainty quantification strategies (such as thresholding the log-likelihood of a response) where hallucinations in the multi-answer case cannot be detected. We conduct a series of experiments which demonstrate the advantage of our formulation. Further, our investigations shed some light on how the probabilities assigned to a given output by an LLM can be amplified by iterative prompting, which might be of independent interest.

Key facts. Cited by 120 papers in the map (data through 7 Oct 2026). Builds on 41 of its 62 references that are in the map. First posted on arXiv 4 Jun 2024.

Builds on

41 of 62 references.

References · 41

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  3. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  4. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  5. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  6. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  7. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  8. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  9. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
  10. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  11. 2023The Internal State of an LLM Knows When It's Lyingcited by 644
  12. 2016Deep Exploration via Bootstrapped DQNcited by 518
29 more
  1. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 385
  2. 2021Entity-Based Knowledge Conflicts in Question Answeringcited by 349
  3. 2024INSIDE: LLMs' Internal States Retain the Power of Hallucination Detectioncited by 317
  4. 2023Generating with Confidence: Uncertainty Quantification for Black-box Large Language Modelscited by 286
  5. 2019Negated and Misprimed Probes for Pretrained Language Models: Birds Can Talk, But Cannot Flycited by 240
  6. 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 209
  7. 2023Universal Self-Consistency for Large Language Model Generationcited by 198
  8. 2022Large Language Models with Controllable Working Memorycited by 169
  9. 2021Epistemic Neural Networkscited by 120
  10. 2023Decomposing Uncertainty for Large Language Models through Input Clarification Ensemblingcited by 113
  11. 2022DisentQA: Disentangling Parametric and Contextual Knowledge with Counterfactual Question Answeringcited by 93
  12. 2023SAC3: Reliable Hallucination Detection in Black-Box Language Models via Semantic-aware Cross-check Consistencycited by 89
  13. 2023Are You Sure? Challenging LLMs Leads to Performance Drops in The FlipFlop Experimentcited by 71
  14. 2023Selectively Answering Ambiguous Questionscited by 63
  15. 2020Depth Uncertainty in Neural Networkscited by 56
  16. 2024Mitigating LLM Hallucinations via Conformal Abstentioncited by 56
  17. 2023Knowing What LLMs DO NOT Know: A Simple Yet Effective Self-Detection Methodcited by 51
  18. 2024Distinguishing the Knowable from the Unknowable with Language Modelscited by 45
  19. 2024Characterizing Truthfulness in Large Language Model Generations with Local Intrinsic Dimensioncited by 45
  20. 2024In-Context Sharpness as Alerts: An Inner Representation Perspective for Hallucination Mitigationcited by 43
  21. 2024Think Twice Before Trusting: Self-Detection for Large Language Models through Comprehensive Answer Reflectioncited by 40
  22. 2023Conformal Nucleus Samplingcited by 32
  23. 2021From Predictions to Decisions: The Importance of Joint Predictive Distributionscited by 28
  24. 2022Selective Prediction via Training Dynamicscited by 20
  25. 2022Ensembles for Uncertainty Estimation: Benefits of Prior Functions and Bootstrappingcited by 19
  26. 2024Narrowing the Knowledge Evaluation Gap: Open-Domain Question Answering with Multi-Granularity Answerscited by 18
  27. 2024Experts Don't Cheat: Learning What You Don't Know By Predicting Pairscited by 18
  28. 2024Understanding the Effects of Iterative Prompting on Truthfulnesscited by 14
  29. 2023On the Intersection of Self-Correction and Trust in Language Modelscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 28; 2025: 55; 2026: 37.

120 papers.

Sort:

By area:

  1. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 115
  2. 2024What Did I Do Wrong? Quantifying LLMs' Sensitivity and Consistency to Prompt Engineeringcited by 88
  3. 2024Learning to Route LLMs with Confidence Tokenscited by 66
  4. 2024Internal Consistency and Self-Feedback in Large Language Models: A Surveycited by 49
  5. 2024BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Modelscited by 48
  6. 2024Knowledge Boundary of Large Language Models: A Surveycited by 36
  7. 2025Trust Me, I'm Wrong: LLMs Hallucinate with Certainty Despite Knowing the Answercited by 26
  8. 2025Understanding the Uncertainty of LLM Explanations: A Perspective Based on Reasoning Topologycited by 24
  9. 2025TokUR: Token-Level Uncertainty Estimation for Large Language Model Reasoningcited by 24
  10. 2024Multilingual Blending: LLM Safety Alignment Evaluation with Language Mixturecited by 22
  11. 2024On Mitigating Code LLM Hallucinations with API Documentationcited by 21
  12. 2026Aligning Language Models from User Interactionscited by 21
38 more
  1. 2025Do We Truly Need So Many Samples? Multi-LLM Repeated Sampling Efficiently Scales Test-Time Computecited by 20
  2. 2024Exploring Response Uncertainty in MLLMs: An Empirical Evaluation under Misleading Scenarioscited by 19
  3. 2025Know What You Don't Know: Uncertainty Calibration of Process Reward Modelscited by 19
  4. 2024Rethinking Uncertainty Estimation in LLMs: A Principled Single-Sequence Measurecited by 18
  5. 2024Meta-Chunking: Learning Text Segmentation and Semantic Completion via Logical Perceptioncited by 17
  6. 2024On Information-Theoretic Measures of Predictive Uncertaintycited by 16
  7. 2024Domain-Specific Retrieval-Augmented Generation Using Vector Stores, Knowledge Graphs, and Tensor Factorizationcited by 15
  8. 2025If Concept Bottlenecks are the Question, are Foundation Models the Answer?cited by 15
  9. 2025MetaFaith: Faithful Natural Language Uncertainty Expression in LLMscited by 15
  10. 2024On Subjective Uncertainty Quantification and Calibration in Natural Language Generationcited by 14
  11. 2025Calibrating Verbalized Confidence with Self-Generated Distractorscited by 14
  12. 2024DebUnc: Improving Large Language Model Agent Communication With Uncertainty Metricscited by 13
  13. 2025LAMARL: LLM-Aided Multi-Agent Reinforcement Learning for Cooperative Policy Generationcited by 13
  14. 2024CALMM-Drive: Confidence-Aware Autonomous Driving with Large Multimodal Modelcited by 12
  15. 2025Reflections on the Reproducibility of Commercial LLM Performance in Empirical Software Engineering Studiescited by 12
  16. 2024Rethinking LLM Uncertainty: A Multi-Agent Approach to Estimating Black-Box Model Uncertaintycited by 11
  17. 2025Do not Abstain! Identify and Solve the Uncertaintycited by 11
  18. 2024Training-Free Bayesianization for Low-Rank Adapters of Large Language Modelscited by 10
  19. 2025Uncertainty Quantification for Hallucination Detection in Large Language Models: Foundations, Methodology, and Future Directionscited by 10
  20. 2024LLMs are not Zero-Shot Reasoners for Biomedical Information Extractioncited by 9
  21. 2025Predicting the Performance of Black-box LLMs through Follow-up Queriescited by 9
  22. 2025C-LoRA: Contextual Low-Rank Adaptation for Uncertainty Estimation in Large Language Modelscited by 9
  23. 2026Complementing Self-Consistency with Cross-Model Disagreement for Uncertainty Quantificationcited by 9
  24. 2024MoRSE: Bridging the Gap in Cybersecurity Expertise with Retrieval Augmented Generationcited by 8
  25. 2025Societal Alignment Frameworks Can Improve LLM Alignmentcited by 8
  26. 2025Textual Bayes: Quantifying Prompt Uncertainty in LLM-Based Systemscited by 8
  27. 2024Defining Knowledge: Bridging Epistemology and Large Language Modelscited by 7
  28. 2025Addressing Pitfalls in the Evaluation of Uncertainty Estimation Methods for Natural Language Generationcited by 7
  29. 2025Uncertainty-Aware Answer Selection for Improved Reasoning in Multi-LLM Systemscited by 6
  30. 2025Uncertainty as Feature Gaps: Epistemic Uncertainty Quantification of LLMs in Contextual Question-Answeringcited by 6
  31. 2025HACK: Hallucinations Along Certainty and Knowledge Axescited by 6
  32. 2025Do Large Language Models (Really) Need Statistical Foundations?cited by 5
  33. 2025Automated Detection of Pre-training Text in Black-box LLMscited by 5
  34. 2025Mind the Generation Process: Fine-Grained Confidence Estimation During LLM Generationcited by 5
  35. 2025Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMscited by 5
  36. 2026Overconfidence and Calibration in Medical VQA: Empirical Findings and Hallucination-Aware Mitigationcited by 5
  37. 2025When an LLM is apprehensive about its answers -- and when its uncertainty is justifiedcited by 4
  38. 2025Hallucination Detection on a Budget: Efficient Bayesian Estimation of Semantic Entropycited by 4

50 most cited of 120 All 120 (JSON)