mlsanityData to 7 Oct 2026

VLM-UQBench: A Benchmark for Modality-Specific and Cross-Modality Uncertainties in Vision Language Models

arXiv 2602.09214 ↗First posted 9 Feb 2026cs.CVcited by 3

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Uncertainty quantification (UQ) is vital for ensuring that vision-language models (VLMs) behave safely and reliably. A central challenge is to localize uncertainty to its source, determining whether it arises from the image, the text, or misalignment between the two. We introduce VLM-UQBench, a benchmark for modality-specific and cross-modal data uncertainty in VLMs, It consists of 600 real-world samples drawn from the VizWiz dataset, curated into clean, image-, text-, and cross-modal uncertainty subsets, and a scalable perturbation pipeline with 8 visual, 5 textual, and 3 cross-modal perturbations. We further propose two simple metrics that quantify the sensitivity of UQ scores to these perturbations and their correlation with hallucinations, and use them to evaluate a range of UQ methods across four VLMs and three datasets. Empirically, we find that: (i) existing UQ methods exhibit strong modality-specific specialization and substantial dependence on the underlying VLM, (ii) modality-specific uncertainty frequently co-occurs with hallucinations while current UQ scores provide only weak and inconsistent risk signals, and (iii) although UQ methods can rival reasoning-based chain-of-thought baselines on overt, group-level ambiguity, they largely fail to detect the subtle, instance-level ambiguity introduced by our perturbation pipeline. These results highlight a significant gap between current UQ practices and the fine-grained, modality-aware uncertainty required for reliable VLM deployment.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 30 of its 52 references that are in the map. First posted on arXiv 9 Feb 2026.

Builds on

30 of 52 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025Probabilistic Embeddings for Frozen Vision-Language Models: Uncertainty Quantification with Gaussian Process Latent Variable Models8 May
  2. 2024VL-Uncertainty: Detecting Hallucination in Large Vision-Language Model via Uncertainty Estimation18 Nov

Other references · 28

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  3. 2024GPT-4o System Cardcited by 4,467
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  5. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  6. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  7. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  8. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  9. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,623
  10. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  11. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  12. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 385
16 more
  1. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 298
  2. 2023Generating with Confidence: Uncertainty Quantification for Black-box Large Language Modelscited by 286
  3. 2024MuirBench: A Comprehensive Benchmark for Robust Multi-image Understandingcited by 178
  4. 2021On Hallucination and Predictive Uncertainty in Conditional Language Generationcited by 176
  5. 2025AbstentionBench: Reasoning LLMs Fail on Unanswerable Questionscited by 129
  6. 2023Decomposing Uncertainty for Large Language Models through Input Clarification Ensemblingcited by 113
  7. 2022CLAM: Selective Clarification for Ambiguous Questions with Generative Language Modelscited by 66
  8. 2023Uncertainty in Natural Language Generation: From Theory to Applicationscited by 61
  9. 2024Uncertainty in Language Models: Assessment through Rank-Calibrationcited by 40
  10. 2024Selective "Selective Prediction": Reducing Unnecessary Abstention in Vision-Language Reasoningcited by 37
  11. 2013Some Improvements on Deep Convolutional Neural Network Based Image Classificationcited by 34
  12. 2025Position: Uncertainty Quantification Needs Reassessment for Large-language Model Agentscited by 27
  13. 2024Certainly Uncertain: A Benchmark and Metric for Multimodal Epistemic and Aleatoric Awarenesscited by 13
  14. 2025Acknowledging Focus Ambiguity in Visual Questionscited by 7
  15. 2024Rethinking the Uncertainty: A Critical Review and Analysis in the Era of Large Language Modelscited by 6
  16. 2024Semantic Consistency-Based Uncertainty Quantification for Factuality in Radiology Report Generationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Walking Through Uncertainty: An Empirical Study of Uncertainty Estimation for Audio-Aware Large Language Modelscited by 4
  2. 2026Don't Blink: Evidence Collapse during Multimodal Reasoningcited by 2
  3. 2026Uncertainty-Aware Decision Making in Multimodal Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Leveraging Visual Signals for Robust Token-Level Uncertainty in Vision-Language Generation
  2. 2024Uncertainty-Aware Evaluation for Vision-Language Models
  3. 2025The Art of Saying "Maybe": A Conformal Lens for Uncertainty Benchmarking in VLMs
  4. 2025Seeing is Believing, but How Much? A Comprehensive Analysis of Verbalized Calibration in Vision-Language Models
  5. 2026FUSE: Quantifying Uncertainty in Vision-Language Models by Bayesian Fusing Epistemic and Aleatoric Uncertainty
  6. 2025Object-Level Verbalized Confidence Calibration in Vision-Language Models via Semantic Perturbation
  7. 2026VAUQ: Vision-Aware Uncertainty Quantification for LVLM Self-Evaluation
  8. 2026VLM-RobustBench: A Comprehensive Benchmark for Robustness of Vision-Language Models