mlsanityData to 14 Oct 2026

Enhancing Vision-Language Model Reliability with Uncertainty-Guided Dropout Decoding

arXiv 2412.06474 ↗First posted 9 Dec 2024cs.CV cs.AI cs.LGcited by 15

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Large vision-language models (LVLMs) excel at multimodal tasks but are prone to misinterpreting visual inputs, often resulting in hallucinations and unreliable outputs. We present DROPOUT DECODING, a novel inference-time approach that quantifies the uncertainty of visual tokens and selectively masks uncertain tokens to improve decoding. Our method measures the uncertainty of each visual token by projecting it onto the text space and decomposing it into aleatoric and epistemic components. Specifically, we focus on epistemic uncertainty, which captures perception-related errors more effectively. Inspired by dropout regularization, we introduce uncertainty-guided token dropout, which applies the dropout principle to input visual tokens instead of model parameters, and during inference rather than training. By aggregating predictions from an ensemble of masked decoding contexts, we can robustly mitigate errors arising from visual token misinterpretations. Evaluations on benchmarks including CHAIR, THRONE, and MMBench demonstrate that DROPOUT DECODING significantly reduces object hallucinations (OH) and enhances both reliability and quality of LVLM outputs across diverse visual contexts. Code is released at https://github.com/kigb/DropoutDecoding.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 59 references that are in the map. First posted on arXiv 9 Dec 2024.

Builds on

32 of 59 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Models4 Oct
  2. 2024Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attention18 Jun
  3. 2024THRONE: An Object-based Hallucination Benchmark for the Free-form Generations of Large Vision-Language Models8 May
  4. 2024A Survey on Hallucination in Large Vision-Language Models1 Feb
  5. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  6. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  7. 2023Detecting and Preventing Hallucinations in Large Vision Language Models11 Aug
  8. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 24

  1. 2023Visual Instruction Tuningcited by 7,869
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  5. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  6. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  7. 2015Show, Attend and Tell: Neural Image Caption Generation with Visual Attentioncited by 1,423
  8. 2018Exploration by Random Network Distillationcited by 993
  9. 2018Object Hallucination in Image Captioningcited by 669
  10. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 623
  11. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  12. 2016Deep Exploration via Bootstrapped DQNcited by 527
12 more
  1. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  2. 2019Deep Ensembles: A Loss Landscape Perspectivecited by 443
  3. 2024Hallucination is Inevitable: An Innate Limitation of Large Language Modelscited by 332
  4. 2024SpatialRGPT: Grounded Spatial Reasoning in Vision Language Modelscited by 311
  5. 2024Enhancing the Reasoning Ability of Multimodal Large Language Models via Mixed Preference Optimizationcited by 239
  6. 2022Factuality Enhanced Language Models for Open-Ended Text Generationcited by 202
  7. 2022A Survey of Vision-Language Pre-Trained Modelscited by 177
  8. 2023Introducing an Improved Information-Theoretic Measure of Predictive Uncertaintycited by 28
  9. 2024Adaptive Contrastive Search: Uncertainty-Guided Decoding for Open-Ended Text Generationcited by 27
  10. 2024Preference Optimization with Multi-Sample Comparisonscited by 16
  11. 2024SLED: Self Logits Evolution Decoding for Improving Factuality in Large Language Modelscited by 16
  12. 2024Direct Acquisition Optimization for Low-Budget Active Learningcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 8.

15 papers.

Sort:

By area:

  1. 2025Seeing and Reasoning with Confidence: Supercharging Multimodal LLMs with an Uncertainty-Aware Agentic Frameworkcited by 12
  2. 2025Text or Pixels? It Takes Half: On the Token Efficiency of Visual Text Inputs in Multimodal LLMscited by 8
  3. 2026Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?cited by 6
  4. 2025Uncertainty-o: One Model-agnostic Framework for Unveiling Uncertainty in Large Multimodal Modelscited by 5
  5. 2026SCALE: Self-uncertainty Conditioned Adaptive Looking and Execution for Vision-Language-Action Modelscited by 4
  6. 2025Entropy Heat-Mapping: Localizing GPT-Based OCR Errors with Sliding-Window Shannon Analysiscited by 2
  7. 2025Mapping User Trust in Vision Language Models: Research Landscape, Challenges, and Prospectscited by 1
  8. 2026SAKED: Mitigating Hallucination in Large Vision-Language Models via Stability-Aware Knowledge Enhanced Decodingcited by 1
  9. 2025Mitigating Object Hallucination via Robust Local Perception Searchcited by 0
  10. 2025Trust in Vision-Language Models: Insights from a Participatory User Workshopcited by 0
  11. 2026Evaluating Reasoning Fidelity in Visual Text Generationcited by 0
  12. 2026Uncertainty-Aware Decision Making in Multimodal Large Language Modelscited by 0
3 more
  1. 2026CUSP: Decomposable Collective Uncertainty for Multi-Agent Multimodal Reasoningcited by 0
  2. 2026Dissecting Training-Free Uncertainty Estimation in Multimodal Large Language Modelscited by 0
  3. 2026ROT: Rotating Hidden States towards Contextual Vectors for Hallucination Mitigation in LVLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Toward More Reliable Artificial Intelligence: Reducing Hallucinations in Vision-Language Models
  2. 2025On Epistemic Uncertainty of Visual Tokens for Object Hallucinations in Large Vision-Language Models
  3. 2024VaLiD: Mitigating the Hallucination of Large Vision Language Models by Visual Layer Fusion Contrastive Decoding
  4. 2025Revisit What You See: Revealing Visual Semantics in Vision Tokens to Guide LVLM Decoding
  5. 2026Leveraging Visual Signals for Robust Token-Level Uncertainty in Vision-Language Generation
  6. 2025Curing Semantic Drift: A Dynamic Approach to Grounding Generation in Large Vision-Language Models
  7. 2025Extracting Visual Facts from Intermediate Layers for Mitigating Hallucinations in Multimodal Large Language Models
  8. 2026FUSE: Quantifying Uncertainty in Vision-Language Models by Bayesian Fusing Epistemic and Aleatoric Uncertainty