# Beyond Semantic Entropy: Boosting LLM Uncertainty Quantification with Pairwise Semantic Similarity

arXiv: https://arxiv.org/abs/2506.00245 · HTML: https://mlsanity.com/paper/2506.00245/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 37 references that are in the map.
- First posted on arXiv 30 May 2025.

## Builds on (most cited first, 23 of 23)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [DeBERTa: Decoding-enhanced BERT with Disentangled Attention](https://mlsanity.com/paper/2006.03654/) (2020)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [Are NLP Models really able to Solve Simple Math Word Problems?](https://mlsanity.com/paper/2103.07191/) (2021)
- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)
- [How Good Are GPT Models at Machine Translation? A Comprehensive Evaluation](https://mlsanity.com/paper/2302.09210/) (2023)
- [Unsupervised Quality Estimation for Neural Machine Translation](https://mlsanity.com/paper/2005.10608/) (2020)
- [Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similarities](https://mlsanity.com/paper/2405.20003/) (2024)
- [LM-Polygraph: Uncertainty Estimation for Language Models](https://mlsanity.com/paper/2311.07383/) (2023)
- [Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraph](https://mlsanity.com/paper/2406.15627/) (2024)
- [LUQ: Long-text Uncertainty Quantification for LLMs](https://mlsanity.com/paper/2403.20279/) (2024)
- [This Email Could Save Your Life: Introducing the Task of Email Subject Line Generation](https://mlsanity.com/paper/1906.03497/) (2019)
- [A Survey of Uncertainty Estimation in LLMs: Theory Meets Practice](https://mlsanity.com/paper/2410.15326/) (2024)

## Built on by (most cited first, 28 of 28)

- [Trustworthy Summarization via Uncertainty Quantification and Risk Awareness in Large Language Models](https://mlsanity.com/paper/2510.01231/) (2025)
- [Estimating Semantic Alphabet Size for LLM Uncertainty Quantification](https://mlsanity.com/paper/2509.14478/) (2025)
- [Semantic Uncertainty Quantification of Hallucinations in LLMs: A Quantum Tensor Network Based Method](https://mlsanity.com/paper/2601.20026/) (2026)
- [Enhancing Multi-Modal LLMs Reasoning via Difficulty-Aware Group Normalization](https://mlsanity.com/paper/2602.21743/) (2026)
- [Walking Through Uncertainty: An Empirical Study of Uncertainty Estimation for Audio-Aware Large Language Models](https://mlsanity.com/paper/2604.25591/) (2026)
- [When to Trust the Answer: Question-Aligned Semantic Nearest Neighbor Entropy for Safer Surgical VQA](https://mlsanity.com/paper/2511.01458/) (2025)
- [Improving Semantic Uncertainty Quantification in LVLMs with Semantic Gaussian Processes](https://mlsanity.com/paper/2512.14177/) (2025)
- [Concept than Document: Context Compression via AMR-based Conceptual Entropy](https://mlsanity.com/paper/2511.18832/) (2025)
- [Dep-LLM: Training-Free Depression Diagnosis via Evidence-Guided Structured Multi-factor with Reliable LLM Reasoning](https://mlsanity.com/paper/2606.10796/) (2026)
- [A Systematic Evaluation of Black-Box Uncertainty Estimation Methods for Large Language Models](https://mlsanity.com/paper/2606.19868/) (2026)
- [TriLens: Per-Layer Logit-Lens Entropy for White-Box Hallucination Detection](https://mlsanity.com/paper/2606.01033/) (2026)
- [When Does Delegation Beat Majority? A Delegation-Based Aggregator for Multi-Sample LLM Inference](https://mlsanity.com/paper/2606.08098/) (2026)
- [Navigating Unreliable Parametric and Contextual Knowledge: Explicit Knowledge Conflict Resolution for LLM Inference](https://mlsanity.com/paper/2606.20245/) (2026)
- [Trust Your Instincts: Confidence-Driven Test-Time RL for Vision-Language-Action Models](https://mlsanity.com/paper/2606.29892/) (2026)
- [Mind the Unseen Mass: Unmasking LLM Hallucinations via Soft-Hybrid Alphabet Estimation](https://mlsanity.com/paper/2604.19162/) (2026)
- [Position: Uncertainty Quantification in LLMs is Just Unsupervised Clustering](https://mlsanity.com/paper/2605.19220/) (2026)
- [Leveraging Visual Signals for Robust Token-Level Uncertainty in Vision-Language Generation](https://mlsanity.com/paper/2605.27136/) (2026)
- [Visual Semantic Entropy: Do Vision Language Models Recognize Visual Ambiguity?](https://mlsanity.com/paper/2606.31407/) (2026)
- [Evaluation Pitfalls and Sparsity Limitations in LLM-based Confidence Estimates for Classification](https://mlsanity.com/paper/2608.04899/) (2026)
- [Decomposing Wrong-Consensus Agreement in LLM Self-Consistency](https://mlsanity.com/paper/2608.18795/) (2026)
- [Kernel Token Contradiction: a Fast and Principled Approach for LLM Claim Uncertainty Quantification](https://mlsanity.com/paper/2608.22506/) (2026)
- [BiG-SURE - Bipartite Graph for Semantic Uncertainty and Reliability Estimation of LLMs](https://mlsanity.com/paper/2608.30646/) (2026)
- [From Tokens to Semantics: Leveraging Complementary Signals for Hallucination Detection in Black-Box LLMs](https://mlsanity.com/paper/2609.02679/) (2026)
- [Using Semantic Uncertainty to Estimate Transition Relevance in Turn-taking](https://mlsanity.com/paper/2609.10934/) (2026)
- [Semantic Uncertainty Quantification Needs Factual Equivalence](https://mlsanity.com/paper/2609.34967/) (2026)
- ["very likely" Means "uncertain"? How LLMs Diverge from Humans in Linguistic Uncertainty Quantification](https://mlsanity.com/paper/2610.00083/) (2026)
- [Embedding Graph Kernel Entropy (EGKE): Fine-grained Hallucination Detection for LLMs from Embeddings](https://mlsanity.com/paper/2610.14214/) (2026)
- [Rarity Determines Minimax Rates of Semantic Entropy From Weak Pairwise Judgments With a Known Channel](https://mlsanity.com/paper/2610.17971/) (2026)

## Abstract (arXiv)

Hallucination in large language models (LLMs) can be detected by assessing the uncertainty of model outputs, typically measured using entropy. Semantic entropy (SE) enhances traditional entropy estimation by quantifying uncertainty at the semantic cluster level. However, as modern LLMs generate longer one-sentence responses, SE becomes less effective because it overlooks two crucial factors: intra-cluster similarity (the spread within a cluster) and inter-cluster similarity (the distance between clusters). To address these limitations, we propose a simple black-box uncertainty quantification method inspired by nearest neighbor estimates of entropy. Our approach can also be easily extended to white-box settings by incorporating token probabilities. Additionally, we provide theoretical results showing that our method generalizes semantic entropy. Extensive empirical results demonstrate its effectiveness compared to semantic entropy across two recent LLMs (Phi3 and Llama3) and three common text generation tasks: question answering, text summarization, and machine translation. Our code is available at https://github.com/BigML-CS-UCLA/SNNE.