mlsanityData to 14 Oct 2026

FUSE: Quantifying Uncertainty in Vision-Language Models by Bayesian Fusing Epistemic and Aleatoric Uncertainty

arXiv 2606.14728 ↗First posted 1 Jun 2026cs.CVcited by 1

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Vision-language models (VLMs) are playing an increasingly important role across multiple domains. In many applications, such as robotics, it is crucial to quantify the uncertainty in the output of these models. } We develop FUSE, a probabilistic framework for capturing two complementary sources of uncertainty in vision-language modeling: (i) aleatoric embedding-level uncertainty derived from input data vision-language ambiguity, and (ii) epistemic model-level uncertainty estimated from the semantic response diversity of VLMs. Our approach formulates a Bayesian fusion mechanism that analytically combines these uncertainty sources to produce a scalar measure of uncertainty. This measure can be used to reliably predict the model's output correctness for downstream applications. We demonstrate that our method outperforms baselines and achieves SOTA uncertainty calibration.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 61 of its 78 references that are in the map. First posted on arXiv 1 Jun 2026.

Builds on

61 of 78 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025Probabilistic Embeddings for Frozen Vision-Language Models: Uncertainty Quantification with Gaussian Process Latent Variable Models8 May
  2. 2024VL-Uncertainty: Detecting Hallucination in Large Vision-Language Model via Uncertainty Estimation18 Nov
  3. 2024Hallucination of Multimodal Large Language Models: A Survey29 Apr
  4. 2023Mitigating Fine-Grained Hallucination by Fine-Tuning Large Vision-Language Models with Caption Rewrites4 Dec
  5. 2023HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Data22 Nov

Other references · 56

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2017On Calibration of Modern Neural Networkscited by 4,960
  7. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  10. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  11. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  12. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
38 more
  1. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  2. 2023Qwen Technical Reportcited by 3,344
  3. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  6. 2015VQA: Visual Question Answeringcited by 2,506
  7. 2017What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?cited by 2,352
  8. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  9. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  10. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  11. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  12. 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 913
  13. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 862
  14. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 819
  15. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 810
  16. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  17. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 792
  18. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  19. 2018Evidential Deep Learning to Quantify Classification Uncertaintycited by 732
  20. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  21. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 584
  22. 2022Open-Vocabulary Semantic Segmentation with Mask-adapted CLIPcited by 565
  23. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  24. 2022Conformal prediction beyond exchangeabilitycited by 355
  25. 2024INSIDE: LLMs' Internal States Retain the Power of Hallucination Detectioncited by 323
  26. 2023Generating with Confidence: Uncertainty Quantification for Black-box Large Language Modelscited by 293
  27. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 288
  28. 2021Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Controlcited by 271
  29. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agentscited by 193
  30. 2023Conformal Language Modelingcited by 170
  31. 2024MM-Embed: Universal Multimodal Retrieval with Multimodal LLMscited by 161
  32. 2024Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similaritiescited by 159
  33. 2022Plex: Towards Reliability using Pretrained Large Model Extensionscited by 102
  34. 2023Batch Calibration: Rethinking Calibration for In-Context Learning and Prompt Engineeringcited by 94
  35. 2025Describe Anything: Detailed Localized Image and Video Captioningcited by 84
  36. 2021Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Modelscited by 75
  37. 2024API Is Enough: Conformal Prediction for Large Language Models Without Logit-Accesscited by 69
  38. 2019Challenges in Markov chain Monte Carlo for Bayesian neural networkscited by 44

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Hierarchical Aggregation of Semantic Uncertainty in 3D Scene Graphscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Uncertainty-Aware Evaluation for Vision-Language Models
  2. 2026Leveraging Visual Signals for Robust Token-Level Uncertainty in Vision-Language Generation
  3. 2025The Art of Saying "Maybe": A Conformal Lens for Uncertainty Benchmarking in VLMs
  4. 2026VLM-UQBench: A Benchmark for Modality-Specific and Cross-Modality Uncertainties in Vision Language Models
  5. 2025Exploiting the Asymmetric Uncertainty Structure of Pre-trained VLMs on the Unit Hypersphere
  6. 2025Seeing is Believing, but How Much? A Comprehensive Analysis of Verbalized Calibration in Vision-Language Models
  7. 2026Dynamic Distribution-Aware Uncertainty Tracking in Vision-Language Representation Learning
  8. 2024Enhancing Vision-Language Model Reliability with Uncertainty-Guided Dropout Decoding