mlsanityData to 7 Oct 2026

Visual Semantic Entropy: Do Vision Language Models Recognize Visual Ambiguity?

arXiv 2606.31407 ↗First posted 30 Jun 2026cs.CV cs.AI cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Vision-language models can produce confident answers on visually ambiguous inputs, resulting in biased predictions. Common entropy-based methods, such as Semantic Entropy (SE), rely on output diversity. Yet our analysis shows that overconfident visual embeddings suppress output diversity under stochastic decoding, causing SE to underestimate uncertainty in such cases. Recent methods instead probe output diversity through input perturbations, including textual paraphrasing or joint text-image perturbations, and show improved performance. We study these approaches and reveals that the resulting variability is often dominated by textual changes rather than visual evidence, causing uncertainty estimates to reflect prompt sensitivity rather than visual ambiguity. We therefore propose Visual Semantic Entropy (VSE), which perturbs only the image to probe nearby visual variations while keeping the text query fixed. VSE measures uncertainty by clustering generated answers into semantic prototypes and computing the mass-weighted dispersion among them. Extensive evaluation across five modern vision-language models and five diverse VQA benchmarks demonstrates that VSE effectively captures visual ambiguity, establishing a new state-of-the-art for VLM uncertainty estimation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 29 references that are in the map. First posted on arXiv 30 Jun 2026.

Builds on

24 of 29 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Seeing is Believing, but How Much? A Comprehensive Analysis of Verbalized Calibration in Vision-Language Models26 May
  2. 2024VL-Uncertainty: Detecting Hallucination in Large Vision-Language Model via Uncertainty Estimation18 Nov
  3. 2024RadFlag: A Black-Box Hallucination Detection Method for Medical Vision Language Models1 Nov
  4. 2024Reference-free Hallucination Detection for Large Vision-Language Models11 Aug
  5. 2024Overconfidence is Key: Verbalized Uncertainty Evaluation in Large Language and Vision-Language Models5 May

Other references · 19

  1. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  2. 2025Qwen2.5-VL Technical Reportcited by 5,418
  3. 2025Qwen3-VL Technical Reportcited by 2,534
  4. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  5. 2025Gemma 3 Technical Reportcited by 1,686
  6. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  7. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  8. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  9. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  10. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
  11. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  12. 2026Qwen3-VL-Embedding and Qwen3-VL-Reranker: A Unified Framework for State-of-the-Art Multimodal Retrieval and Rankingcited by 211
7 more
  1. 2024Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similaritiescited by 156
  2. 2025Vision Language Models are Biasedcited by 63
  3. 2017Hierarchical Clustering: Objective Functions and Algorithmscited by 60
  4. 2024Probing Visual Language Priors in VLMscited by 44
  5. 2024Consistency and Uncertainty: Identifying Unreliable Responses From Black-Box Vision-Language Models for Selective Visual Question Answeringcited by 31
  6. 2025Beyond Semantic Entropy: Boosting LLM Uncertainty Quantification with Pairwise Semantic Similaritycited by 26
  7. 2021I Find Your Lack of Uncertainty in Computer Vision Disturbingcited by 16

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Diagnosing Visual Ignorance in Vision-Language Models
  2. 2026Leveraging Visual Signals for Robust Token-Level Uncertainty in Vision-Language Generation
  3. 2026FUSE: Quantifying Uncertainty in Vision-Language Models by Bayesian Fusing Epistemic and Aleatoric Uncertainty
  4. 2025Exploiting the Asymmetric Uncertainty Structure of Pre-trained VLMs on the Unit Hypersphere
  5. 2024Uncertainty-Aware Evaluation for Vision-Language Models
  6. 2025Mixed Signals: Decoding VLMs' Reasoning and Underlying Bias in Vision-Language Conflict
  7. 2023Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding
  8. 2024Enhancing Vision-Language Model Reliability with Uncertainty-Guided Dropout Decoding