mlsanityData to 7 Oct 2026

Multi-Modal Hallucination Control by Visual Information Grounding

arXiv 2403.14003 ↗First posted 20 Mar 2024cs.CV cs.CL cs.LGcited by 196

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

also in 1 more
Abstract (arXiv)

Generative Vision-Language Models (VLMs) are prone to generate plausible-sounding textual answers that, however, are not always grounded in the input image. We investigate this phenomenon, usually referred to as "hallucination" and show that it stems from an excessive reliance on the language prior. In particular, we show that as more tokens are generated, the reliance on the visual prompt decreases, and this behavior strongly correlates with the emergence of hallucinations. To reduce hallucinations, we introduce Multi-Modal Mutual-Information Decoding (M3ID), a new sampling method for prompt amplification. M3ID amplifies the influence of the reference image over the language prior, hence favoring the generation of tokens with higher mutual information with the visual prompt. M3ID can be applied to any pre-trained autoregressive VLM at inference time without necessitating further training and with minimal computational overhead. If training is an option, we show that M3ID can be paired with Direct Preference Optimization (DPO) to improve the model's reliance on the prompt image without requiring any labels. Our empirical findings show that our algorithms maintain the fluency and linguistic capabilities of pre-trained VLMs while reducing hallucinations by mitigating visually ungrounded answers. Specifically, for the LLaVA 13B model, M3ID and M3ID+DPO reduce the percentage of hallucinated objects in captioning tasks by 25% and 28%, respectively, and improve the accuracy on VQA benchmarks such as POPE by 21% and 24%.

Key facts. Cited by 196 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 25 of its 32 references that are in the map. First posted on arXiv 20 Mar 2024.

Builds on

25 of 32 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
  2. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 23

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  12. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
11 more
  1. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  2. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  3. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  4. 2018Object Hallucination in Image Captioningcited by 661
  5. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  6. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 319
  7. 2022Locally Typical Samplingcited by 108
  8. 2021The Factual Inconsistency Problem in Abstractive Text Summarization: A Surveycited by 79
  9. 2022Mutual Information Alleviates Hallucinations in Abstractive Summarizationcited by 67
  10. 2023Pointwise Mutual Information Based Metric and Decoding Strategy for Faithful Generation in Document Grounded Dialogscited by 11
  11. 2021A novel Deep Neural Network architecture for non-linear system identificationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 31; 2025: 85; 2026: 80.

196 papers.

Sort:

By area:

  1. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  2. 2025OpenDriveVLA: Towards End-to-end Autonomous Driving with Large Vision Language Action Modelcited by 172
  3. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lenscited by 139
  4. 2024RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthinesscited by 129
  5. 2025See What You Are Told: Visual Attention Sink in Large Multimodal Modelscited by 127
  6. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Modelscited by 90
  7. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Modelcited by 73
  8. 2025The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models via Visual Information Steeringcited by 62
  9. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steeringcited by 56
  10. 2024Visual Prompting in Multimodal Large Language Models: A Surveycited by 52
  11. 2024Mitigating Object Hallucination via Concentric Causal Attentioncited by 48
  12. 2025UAVs Meet LLMs: Overviews and Perspectives Toward Agentic Low-Altitude Mobilitycited by 48
38 more
  1. 2025Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMscited by 48
  2. 2024DAMRO: Dive into the Attention Mechanism of LVLM to Reduce Object Hallucinationcited by 47
  3. 2024Don't Miss the Forest for the Trees: Attentional Vision Calibration for Large Vision Language Modelscited by 46
  4. 2024Nullu: Mitigating Object Hallucinations in Large Vision-Language Models via HalluSpace Projectioncited by 46
  5. 2025Self-Correcting Decoding with Generative Feedback for Mitigating Hallucinations in Large Vision-Language Modelscited by 44
  6. 2025Seeing Far and Clearly: Mitigating Hallucinations in MLLMs with Attention Causal Decodingcited by 42
  7. 2024Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimizationcited by 34
  8. 2025Treble Counterfactual VLMs: A Causal Approach to Hallucinationcited by 33
  9. 2025Large Vision-Language Model Alignment and Misalignment: A Survey Through the Lens of Explainabilitycited by 28
  10. 2024B'MOJO: Hybrid State Space Realizations of Foundation Models with Eidetic and Fading Memorycited by 26
  11. 2024Modality-Fair Preference Optimization for Trustworthy MLLM Alignmentcited by 26
  12. 2024Mitigating Hallucinations in Large Vision-Language Models (LVLMs) via Language-Contrastive Decoding (LCD)cited by 23
  13. 2024Unraveling and Mitigating Safety Alignment Degradation of Vision-Language Modelscited by 23
  14. 2025Octopus: Alleviating Hallucination via Dynamic Contrastive Decodingcited by 23
  15. 2026Visual-Advantage On-Policy Distillation for Vision-Language Modelscited by 22
  16. 2024Mitigating Hallucinations in Large Vision-Language Models via Summary-Guided Decodingcited by 21
  17. 2025Cross-Modal Attention Calibration for LVLM Hallucination Mitigationcited by 21
  18. 2025Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in Multimodal Large Language Modelscited by 21
  19. 2024VidHal: Benchmarking Temporal Hallucinations in Vision LLMscited by 19
  20. 2025A Survey on Efficient Vision-Language Modelscited by 19
  21. 2025Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMscited by 16
  22. 2024Pensieve: Retrospect-then-Compare Mitigates Visual Hallucinationcited by 14
  23. 2024Losing Visual Needles in Image Haystacks: Vision Language Models are Easily Distracted in Short and Long Contextscited by 14
  24. 2025Hallucinatory Image Tokens: A Training-free EAZY Approach on Detecting and Mitigating Object Hallucinations in LVLMscited by 14
  25. 2025Localizing Before Answering: A Hallucination Evaluation Benchmark for Grounded Medical Multimodal LLMscited by 14
  26. 2024Certainly Uncertain: A Benchmark and Metric for Multimodal Epistemic and Aleatoric Awarenesscited by 13
  27. 2024ScVLM: Enhancing Vision-Language Model for Safety-Critical Event Understandingcited by 13
  28. 2025Understanding Language Prior of LVLMs by Contrasting Chain-of-Embeddingcited by 13
  29. 2024VaLiD: Mitigating the Hallucination of Large Vision Language Models by Visual Layer Fusion Contrastive Decodingcited by 12
  30. 2024Exploring How Generative MLLMs Perceive More Than CLIP with the Same Vision Encodercited by 11
  31. 2025GLSim: Detecting Object Hallucinations in LVLMs via Global-Local Similaritycited by 11
  32. 2025MedHEval: Benchmarking Hallucinations and Mitigation Strategies in Medical Large Vision-Language Modelscited by 10
  33. 2025Through the Magnifying Glass: Adaptive Perception Magnification for Hallucination-Free VLM Decodingcited by 10
  34. 2025Mitigating Hallucination in Large Vision-Language Models via Adaptive Attention Calibrationcited by 10
  35. 2025INTER: Mitigating Hallucination in Large Vision-Language Models by Interaction Guidance Samplingcited by 10
  36. 2025Mitigating Hallucination in Multimodal LLMs with Layer Contrastive Decodingcited by 10
  37. 2024Efficient Self-Improvement in Multimodal Large Language Models: A Model-Level Judge-Free Approachcited by 9
  38. 2025Poison as Cure: Visual Noise for Mitigating Object Hallucinations in LVMscited by 9

50 most cited of 196 All 196 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding
  2. 2025When Images Speak Louder: Mitigating Language Bias-induced Hallucinations in VLMs through Cross-Modal Guidance
  3. 2025ONLY: One-Layer Intervention Sufficiently Mitigates Hallucinations in Large Vision-Language Models
  4. 2024Look, Compare, Decide: Alleviating Hallucination in Large Vision-Language Models via Multi-View Multi-Path Reasoning
  5. 2025Exposing Hallucinations To Suppress Them: VLMs Representation Editing With Generative Anchors
  6. 2025OViP: Online Vision-Language Preference Learning for VLM Hallucination
  7. 2026Grounded Post-Training with Hard Examples for Reducing Hallucination in Multimodal Large Language Models
  8. 2026SAGE: Sink-Aware Grounded Decoding for Multimodal Hallucination Mitigation