mlsanityData to 7 Oct 2026

LLM Uncertainty Quantification through Directional Entailment Graph and Claim Level Response Augmentation

arXiv 2407.00994 ↗First posted 1 Jul 2024cs.CLcited by 22

Not on the map. Why?

Abstract (arXiv)

The Large language models (LLMs) have showcased superior capabilities in sophisticated tasks across various domains, stemming from basic question-answer (QA), they are nowadays used as decision assistants or explainers for unfamiliar content. However, they are not always correct due to the data sparsity in specific domain corpus, or the model's hallucination problems. Given this, how much should we trust the responses from LLMs? This paper presents a novel way to evaluate the uncertainty that captures the directional instability, by constructing a directional graph from entailment probabilities, and we innovatively conduct Random Walk Laplacian given the asymmetric property of a constructed directed graph, then the uncertainty is aggregated by the derived eigenvalues from the Laplacian process. We also provide a way to incorporate the existing work's semantics uncertainty with our proposed layer. Besides, this paper identifies the vagueness issues in the raw response set and proposes an augmentation approach to mitigate such a problem, we conducted extensive empirical experiments and demonstrated the superiority of our proposed solutions.

Key facts. Cited by 22 papers in the map (data through 7 Oct 2026). Builds on 17 of its 31 references that are in the map. First posted on arXiv 1 Jul 2024.

Builds on

17 of 31 references.

References · 17

  1. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  2. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  3. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  4. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  5. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  6. 2020A Review of Uncertainty Quantification in Deep Learning: Techniques, Applications and Challengescited by 813
  7. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  8. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  9. 2018CoQA: A Conversational Question Answering Challengecited by 674
  10. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  11. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  12. 2024LLMs Can't Plan, But Can Help Planning in LLM-Modulo Frameworkscited by 178
5 more
  1. 2023Quantifying Uncertainty in Answers from any Language Model and Enhancing their Trustworthinesscited by 121
  2. 2024Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvementcited by 74
  3. 2023Open-TI: Open Traffic Intelligence with Augmented Language Modelcited by 38
  4. 2023Prompt to Transfer: Sim-to-Real Transfer for Traffic Signal Control with Prompt Learningcited by 36
  5. 2024FACT-GPT: Fact-Checking Augmentation via Claim Matching with LLMscited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 12; 2026: 9.

22 papers.

Sort:

By area:

  1. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 115
  2. 2025Uncertainty Quantification and Confidence Calibration in Large Language Models: A Surveycited by 93
  3. 2025A Survey of Sim-to-Real Methods in RL: Progress, Prospects and Challenges with Foundation Modelscited by 36
  4. 2025Understanding the Uncertainty of LLM Explanations: A Perspective Based on Reasoning Topologycited by 24
  5. 2025Uncertainty Quantification of Large Language Models through Multi-Dimensional Responsescited by 17
  6. 2025Uncertainty Quantification for Hallucination Detection in Large Language Models: Foundations, Methodology, and Future Directionscited by 10
  7. 2025Generative AI in Transportation Planning: A Surveycited by 8
  8. 2026Every Response Counts: Quantifying Uncertainty of LLM-based Multi-Agent Systems through Tensor Decompositioncited by 5
  9. 2026Are LLM Uncertainty and Correctness Encoded by the Same Features? A Functional Dissociation via Sparse Autoencoderscited by 5
  10. 2025MCQA-Eval: Efficient Confidence Evaluation in NLG with Gold-Standard Correctness Labelscited by 4
  11. 2025TruthTorchLM: A Comprehensive Library for Predicting Truthfulness in LLM Outputscited by 4
  12. 2026Conformal Feedback Alignment: Quantifying Answer-Level Reliability for Robust LLM Alignmentcited by 4
10 more
  1. 2026Fine-Grained Uncertainty Quantification for Long-Form Language Model Outputs: A Comparative Studycited by 4
  2. 2025ConfAgents: A Conformal-Guided Multi-Agent Framework for Cost-Efficient Medical Diagnosiscited by 2
  3. 2025SIMBA UQ: Similarity-Based Aggregation for Uncertainty Quantification in Large Language Modelscited by 2
  4. 2025ALARM: Automated MLLM-Based Anomaly Detection in Complex-EnviRonment Monitoring with Uncertainty Quantificationcited by 2
  5. 2026Diagnosing Multi-step Reasoning Failures in Black-box LLMs via Stepwise Confidence Attributioncited by 2
  6. 2025GE-Chat: A Graph Enhanced RAG Framework for Evidential Response Generation of LLMscited by 1
  7. 2026The Sim-to-Real Gap of Foundation Model Agents: A Unified MDP Perspectivecited by 1
  8. 2026Fact-Checking with Large Language Models via Probabilistic Certainty and Consistencycited by 0
  9. 2026Position: Uncertainty Quantification in LLMs is Just Unsupervised Clusteringcited by 0
  10. 2026SpanUQ: Span-Level Uncertainty Quantification for Large Language Model Generationcited by 0