mlsanityData to 14 Oct 2026

Beyond Semantic Entropy: Boosting LLM Uncertainty Quantification with Pairwise Semantic Similarity

arXiv 2506.00245 ↗First posted 30 May 2025cs.LG cs.CLcited by 28

Not on the map. Why?

Abstract (arXiv)

Hallucination in large language models (LLMs) can be detected by assessing the uncertainty of model outputs, typically measured using entropy. Semantic entropy (SE) enhances traditional entropy estimation by quantifying uncertainty at the semantic cluster level. However, as modern LLMs generate longer one-sentence responses, SE becomes less effective because it overlooks two crucial factors: intra-cluster similarity (the spread within a cluster) and inter-cluster similarity (the distance between clusters). To address these limitations, we propose a simple black-box uncertainty quantification method inspired by nearest neighbor estimates of entropy. Our approach can also be easily extended to white-box settings by incorporating token probabilities. Additionally, we provide theoretical results showing that our method generalizes semantic entropy. Extensive empirical results demonstrate its effectiveness compared to semantic entropy across two recent LLMs (Phi3 and Llama3) and three common text generation tasks: question answering, text summarization, and machine translation. Our code is available at https://github.com/BigML-CS-UCLA/SNNE.

Key facts. Cited by 28 papers in the map (data through 14 Oct 2026). Builds on 23 of its 37 references that are in the map. First posted on arXiv 30 May 2025.

Builds on

23 of 37 references.

References · 23

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  6. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  7. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  8. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  9. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  10. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  11. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  12. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
11 more
  1. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,102
  2. 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
  3. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 723
  4. 2023How Good Are GPT Models at Machine Translation? A Comprehensive Evaluationcited by 348
  5. 2020Unsupervised Quality Estimation for Neural Machine Translationcited by 211
  6. 2024Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similaritiescited by 159
  7. 2023LM-Polygraph: Uncertainty Estimation for Language Modelscited by 131
  8. 2024Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraphcited by 105
  9. 2024LUQ: Long-text Uncertainty Quantification for LLMscited by 81
  10. 2019This Email Could Save Your Life: Introducing the Task of Email Subject Line Generationcited by 58
  11. 2024A Survey of Uncertainty Estimation in LLMs: Theory Meets Practicecited by 55

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 23.

28 papers.

Sort:

By area:

  1. 2025Trustworthy Summarization via Uncertainty Quantification and Risk Awareness in Large Language Modelscited by 10
  2. 2025Estimating Semantic Alphabet Size for LLM Uncertainty Quantificationcited by 8
  3. 2026Semantic Uncertainty Quantification of Hallucinations in LLMs: A Quantum Tensor Network Based Methodcited by 7
  4. 2026Enhancing Multi-Modal LLMs Reasoning via Difficulty-Aware Group Normalizationcited by 4
  5. 2026Walking Through Uncertainty: An Empirical Study of Uncertainty Estimation for Audio-Aware Large Language Modelscited by 4
  6. 2025When to Trust the Answer: Question-Aligned Semantic Nearest Neighbor Entropy for Safer Surgical VQAcited by 3
  7. 2025Improving Semantic Uncertainty Quantification in LVLMs with Semantic Gaussian Processescited by 3
  8. 2025Concept than Document: Context Compression via AMR-based Conceptual Entropycited by 2
  9. 2026Dep-LLM: Training-Free Depression Diagnosis via Evidence-Guided Structured Multi-factor with Reliable LLM Reasoningcited by 2
  10. 2026A Systematic Evaluation of Black-Box Uncertainty Estimation Methods for Large Language Modelscited by 2
  11. 2026TriLens: Per-Layer Logit-Lens Entropy for White-Box Hallucination Detectioncited by 1
  12. 2026When Does Delegation Beat Majority? A Delegation-Based Aggregator for Multi-Sample LLM Inferencecited by 1
16 more
  1. 2026Navigating Unreliable Parametric and Contextual Knowledge: Explicit Knowledge Conflict Resolution for LLM Inferencecited by 1
  2. 2026Trust Your Instincts: Confidence-Driven Test-Time RL for Vision-Language-Action Modelscited by 1
  3. 2026Mind the Unseen Mass: Unmasking LLM Hallucinations via Soft-Hybrid Alphabet Estimationcited by 0
  4. 2026Position: Uncertainty Quantification in LLMs is Just Unsupervised Clusteringcited by 0
  5. 2026Leveraging Visual Signals for Robust Token-Level Uncertainty in Vision-Language Generationcited by 0
  6. 2026Visual Semantic Entropy: Do Vision Language Models Recognize Visual Ambiguity?cited by 0
  7. 2026Evaluation Pitfalls and Sparsity Limitations in LLM-based Confidence Estimates for Classificationcited by 0
  8. 2026Decomposing Wrong-Consensus Agreement in LLM Self-Consistencycited by 0
  9. 2026Kernel Token Contradiction: a Fast and Principled Approach for LLM Claim Uncertainty Quantificationcited by 0
  10. 2026BiG-SURE - Bipartite Graph for Semantic Uncertainty and Reliability Estimation of LLMscited by 0
  11. 2026From Tokens to Semantics: Leveraging Complementary Signals for Hallucination Detection in Black-Box LLMscited by 0
  12. 2026Using Semantic Uncertainty to Estimate Transition Relevance in Turn-takingcited by 0
  13. 2026Semantic Uncertainty Quantification Needs Factual Equivalencecited by 0
  14. 2026"very likely" Means "uncertain"? How LLMs Diverge from Humans in Linguistic Uncertainty Quantificationcited by 0
  15. 2026Embedding Graph Kernel Entropy (EGKE): Fine-grained Hallucination Detection for LLMs from Embeddingscited by 0
  16. 2026Rarity Determines Minimax Rates of Semantic Entropy From Weak Pairwise Judgments With a Known Channelcited by 0