mlsanityData to 14 Oct 2026

Position: Uncertainty Quantification Needs Reassessment for Large-language Model Agents

arXiv 2505.22655 ↗First posted 28 May 2025cs.LG cs.AI cs.CLcited by 32

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Large-language models (LLMs) and chatbot agents are known to provide wrong outputs at times, and it was recently found that this can never be fully prevented. Hence, uncertainty quantification plays a crucial role, aiming to quantify the level of ambiguity in either one overall number or two numbers for aleatoric and epistemic uncertainty. This position paper argues that this traditional dichotomy of uncertainties is too limited for the open and interactive setup that LLM agents operate in when communicating with a user, and that we need to research avenues that enrich uncertainties in this novel scenario. We review the literature and find that popular definitions of aleatoric and epistemic uncertainties directly contradict each other and lose their meaning in interactive LLM agent settings. Hence, we propose three novel research directions that focus on uncertainties in such human-computer interactions: Underspecification uncertainties, for when users do not provide all information or define the exact task at the first go, interactive learning, to ask follow-up questions and reduce the uncertainty about the current context, and output uncertainties, to utilize the rich language and speech space to express uncertainties as more than mere numbers. We expect that these new ways of dealing with and communicating uncertainties will lead to LLM agent interactions that are more transparent, trustworthy, and intuitive.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 43 of its 68 references that are in the map. First posted on arXiv 28 May 2025.

Builds on

43 of 68 references.

References · 43

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,352
  3. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  4. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  5. 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 913
  6. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
  7. 2017Deep Bayesian Active Learning with Image Datacited by 819
  8. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  9. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 390
  10. 2020Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awarenesscited by 389
  11. 2021Deep Deterministic Uncertainty: A Simple Baselinecited by 200
  12. 2023Hallucinations in Large Multilingual Translation Modelscited by 129
31 more
  1. 2019PHiSeg: Capturing Uncertainty in Medical Image Segmentationcited by 128
  2. 2022Quantifying Aleatoric and Epistemic Uncertainty in Machine Learning: Are Conditional Entropy and Mutual Information Appropriate Measures?cited by 125
  3. 2023Calibrated Language Models Must Hallucinatecited by 115
  4. 2024SaySelf: Teaching LLMs to Express Confidence with Self-Reflective Rationalescited by 105
  5. 2021DEUP: Direct Epistemic Uncertainty Predictioncited by 104
  6. 2024Large Language Models Must Be Taught to Know What They Don't Knowcited by 100
  7. 2023Clarify When Necessary: Resolving Ambiguity Through Interaction with LMscited by 98
  8. 2022A Deeper Look into Aleatoric and Epistemic Uncertainty Disentanglementcited by 87
  9. 2024Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words?cited by 87
  10. 2024Benchmarking Uncertainty Disentanglement: Specialized Uncertainties for Specialized Taskscited by 78
  11. 2024LLMs Will Always Hallucinate, and We Need to Live With Thiscited by 78
  12. 2022Pitfalls of Epistemic Uncertainty Quantification through Loss Minimisationcited by 66
  13. 2025Reasoning Models Better Express Their Confidencecited by 65
  14. 2021Building and Evaluating Open-Domain Dialogue Corpora with Clarifying Questionscited by 63
  15. 2023Uncertainty in Natural Language Generation: From Theory to Applicationscited by 61
  16. 2024CLAMBER: A Benchmark of Identifying and Clarifying Ambiguous Information Needs in Large Language Modelscited by 54
  17. 2022Nonparametric Uncertainty Quantification for Single Deterministic Neural Networkcited by 36
  18. 2024Calibrating Long-form Generations from Large Language Modelscited by 32
  19. 2023Probabilistic Contrastive Learning Recovers the Correct Aleatoric Uncertainty of Ambiguous Inputscited by 29
  20. 2024Finetuning Language Models to Emit Linguistic Expressions of Uncertaintycited by 26
  21. 2025CoT-UQ: Improving Response-wise Uncertainty Quantification in LLMs with Chain-of-Thoughtcited by 23
  22. 2021NOMU: Neural Optimization-based Model Uncertaintycited by 16
  23. 2024On Information-Theoretic Measures of Predictive Uncertaintycited by 16
  24. 2024Perceptions of Linguistic Uncertainty by Language Models and Humanscited by 15
  25. 2024(Implicit) Ensembles of Ensembles: Epistemic Uncertainty Collapse in Large Modelscited by 15
  26. 2025UNCLE: Benchmarking Uncertainty Expressions in Long-Form Generationcited by 11
  27. 2022Ensembling over Classifiers: a Bias-Variance Perspectivecited by 6
  28. 2025SelfReflect: Can LLMs Communicate Their Internal Answer Distribution?cited by 6
  29. 2024Measuring Orthogonality as the Blind-Spot of Uncertainty Disentanglementcited by 4
  30. 2024On Uncertainty In Natural Language Processingcited by 3
  31. 2022Towards Clear Expectations for Uncertainty Estimationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 24.

32 papers.

Sort:

By area:

  1. 2025TRiSM for Agentic AI: A Review of Trust, Risk, and Security Management in LLM-based Agentic Multi-Agent Systemscited by 59
  2. 2026Uncertainty Quantification in LLM Agents: Foundations, Emerging Challenges, and Opportunitiescited by 20
  3. 2026Agentic Uncertainty Quantificationcited by 19
  4. 2025The Illusion of Certainty: Uncertainty Quantification for LLMs Fails under Ambiguitycited by 17
  5. 2026Agentic Confidence Calibrationcited by 15
  6. 2026Agentic Abstention: Do Agents Know When to Stop Instead of Act?cited by 15
  7. 2025Anthropomimetic Uncertainty: What Verbalized Uncertainty in Language Models is Missingcited by 14
  8. 2025Uncertainty Quantification for Hallucination Detection in Large Language Models: Foundations, Methodology, and Future Directionscited by 11
  9. 2025Position: Epistemic uncertainty estimation methods are fundamentally incompletecited by 9
  10. 2026The Illusion of Certainty: Decoupling Capability and Calibration in On-Policy Distillationcited by 9
  11. 2026Verbalizing LLM's Higher-order Uncertainty via Imprecise Probabilitiescited by 5
  12. 2025Flow Stochastic Segmentation Networkscited by 3
20 more
  1. 2026VLM-UQBench: A Benchmark for Modality-Specific and Cross-Modality Uncertainties in Vision Language Modelscited by 3
  2. 2026Uncertainty Decomposition for Clarification Seeking in LLM Agentscited by 3
  3. 2025ALARM: Automated MLLM-Based Anomaly Detection in Complex-EnviRonment Monitoring with Uncertainty Quantificationcited by 2
  4. 2025Self-signals Driven Multi-LLM Debate for Efficient and Accurate Reasoningcited by 1
  5. 2026Uncertainty and Fairness Awareness in LLM-Based Recommendation Systemscited by 1
  6. 2026Helicase: Uncertainty-Guided Supply Chain Knowledge Graph Construction with Autonomous Multi-Agent LLMscited by 1
  7. 2026From Sequence to Structure: Relational Uncertainty Propagation for LLM Agentscited by 1
  8. 2026A Vision for Multisensory Intelligence: Sensing, Science, and Synergycited by 0
  9. 2026A Bayesian Perspective on the Role of Epistemic Uncertainty for Delayed Generalization in In-Context Learningcited by 0
  10. 2026Proper Scoring Rules for Agentic Uncertainty Quantificationcited by 0
  11. 2026Position: Prioritize Identifying Structure, Not Complex Models, for Scientific Discoverycited by 0
  12. 2026The safety failures we are not instrumenting: a perspective on hidden safety-critical challenges in modern AI systemscited by 0
  13. 2026An Investigation of the NeurIPS and ICML 2025 Position Trackscited by 0
  14. 2026Improved Confidence Estimates for Black-Box Large Language Modelscited by 0
  15. 2026Uncertainty Quantification for LLM Agents: A Taxonomy, an Evaluation Protocol, and an Empirical Studycited by 0
  16. 2026Prospective Hindsight: Self-Calibrating Reinforcement Learning via Prediction-Reality Gapscited by 0
  17. 2026Single-Pass Uncertainty Heads for Claim-Level Hallucination Detection in Persian Medical Language Modelscited by 0
  18. 2026Uncertainty in Representation Learning on Knowledge Graphscited by 0
  19. 2026Agentic Self-Awareness: Aligning Knowing, Acting, and Holdingcited by 0
  20. 2026What Is a Clarification? Estimating Model-Dependent Input Ambiguitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Rethinking the Uncertainty: A Critical Review and Analysis in the Era of Large Language Models
  2. 2024Towards Uncertainty-Aware Language Agent
  3. 2026CAR-bench: Evaluating the Consistency and Limit-Awareness of LLM Agents under Real-World Uncertainty
  4. 2025UProp: Investigating the Uncertainty Propagation of LLMs in Multi-Step Agentic Decision-Making
  5. 2026From Answers to Interpretations: Rethinking Ambiguity-Induced Aleatoric Uncertainty Estimation in LLMs
  6. 2025Can LLMs Detect Their Confabulations? Estimating Reliability in Uncertainty-Aware Language Models
  7. 2026Uncertainty-Aware Clarification in LLM Agents with Information Gain
  8. 2026Beyond Single-Turn Confidence: Trajectory-Adapted Uncertainty Quantification for LLM Agents