mlsanityData to 7 Oct 2026

Mitigating Object Hallucination in Large Vision-Language Models via Image-Grounded Guidance

arXiv 2402.08680 ↗First posted 13 Feb 2024cs.LG cs.AI cs.CLcited by 44

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

The advancement of Large Vision-Language Models (LVLMs) has increasingly highlighted the critical issue of their tendency to hallucinate non-existing objects in the images. To address this issue, previous works focused on using specially curated datasets or powerful LLMs to rectify the outputs of LVLMs. However, these approaches require either costly training or fine-tuning, or API access to proprietary LLMs for post-generation correction. In response to these limitations, we propose Mitigating hallucinAtion via image-gRounded guIdaNcE (MARINE), a framework that is both training-free and API-free. MARINE effectively and efficiently reduces object hallucinations during inference by introducing image-grounded guidance to LVLMs. This is achieved by leveraging open-source vision models to extract object-level information, thereby enhancing the precision of LVLM-generated content. Our framework's flexibility further allows for the integration of multiple vision models, enabling more reliable and robust object-level guidance. Through comprehensive evaluations across 5 popular LVLMs with diverse evaluation metrics and benchmarks, we demonstrate the effectiveness of MARINE, which even outperforms existing fine-tuning-based methods. Remarkably, it reduces hallucinations consistently in GPT-4V-assisted evaluation while maintaining the detailedness of LVLMs' generations. We release our code at https://github.com/Linxi-ZHAO/MARINE.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 53 of its 77 references that are in the map. First posted on arXiv 13 Feb 2024.

Builds on

53 of 77 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedback1 Dec
  2. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  3. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  4. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Models9 Oct
  5. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
  6. 2023Evaluation and Analysis of Hallucination in Large Vision-Language Models29 Aug
  7. 2023Detecting and Preventing Hallucinations in Large Vision Language Models11 Aug
  8. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
  9. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May
  10. 2022Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-training14 Oct

Other references · 43

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2022Training language models to follow instructions with human feedbackcited by 15,712
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  11. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  12. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
31 more
  1. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  2. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  3. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,624
  4. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  5. 2022Diffusion-LM Improves Controllable Text Generationcited by 1,006
  6. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  7. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  8. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  9. 2018Object Hallucination in Image Captioningcited by 661
  10. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  11. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 416
  12. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217
  13. 2022ReCo: Region-Controlled Text-to-Image Generationcited by 177
  14. 2023Davidsonian Scene Graph: Improving Reliability in Fine-grained Evaluation for Text-to-Image Generationcited by 159
  15. 2022A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Modelscited by 128
  16. 2018Neural Baby Talkcited by 107
  17. 2023Stay on topic with Classifier-Free Guidancecited by 90
  18. 2022Adapting Pretrained Vision-Language Foundational Models to Medical Imaging Domainscited by 85
  19. 2023VIGC: Visual Instruction Generation and Correctioncited by 80
  20. 2022Text Generation with Diffusion Language Models: A Pre-training Approach with Continuous Paragraph Denoisecited by 79
  21. 2020Politeness Transfer: A Tag and Generate Approachcited by 78
  22. 2021Let there be a clock on the beach: Reducing Object Hallucination in Image Captioningcited by 74
  23. 2020End-to-End Object Detection with Fully Convolutional Networkcited by 70
  24. 2023Open-Set Image Tagging with Multi-Grained Text Supervisioncited by 67
  25. 2018Polite Dialogue Generation Without Parallel Datacited by 57
  26. 2020Exploring Controllable Text Generation Techniquescited by 55
  27. 2021Structural Adapters in Pretrained Language Models for AMR-to-text Generationcited by 48
  28. 2022A Causal Lens for Controllable Text Generationcited by 45
  29. 2020The Adapter-Bot: All-In-One Controllable Conversational Modelcited by 38
  30. 2023Prompt Highlighter: Interactive Control for Multi-Modal LLMscited by 29
  31. 2022Aggretriever: A Simple Approach to Aggregate Textual Representations for Robust Dense Passage Retrievalcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 17; 2026: 25.

44 papers.

Sort:

By area:

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  2. 2024Look, Compare, Decide: Alleviating Hallucination in Large Vision-Language Models via Multi-View Multi-Path Reasoningcited by 23
  3. 2025DAMA: Data- and Model-aware Alignment of Multi-modal LLMscited by 14
  4. 2025SEASON: Mitigating Temporal Hallucination in Video Large Language Models via Self-Diagnostic Contrastive Decodingcited by 8
  5. 2026Vision-Language Introspection: Mitigating Overconfident Hallucinations in MLLMs via Interpretable Bi-Causal Steeringcited by 8
  6. 2024RITUAL: Random Image Transformations as a Universal Anti-hallucination Lever in Large Vision Language Modelscited by 7
  7. 2025Mitigating Hallucinations via Inter-Layer Consistency Aggregation in Large Vision-Language Modelscited by 7
  8. 2025SHIELD: Suppressing Hallucinations In LVLM Encoders via Bias and Vulnerability Defensecited by 7
  9. 2025Tell Model Where to Look: Mitigating Hallucinations in MLLMs by Vision-Guided Attentioncited by 6
  10. 2025Losing the Plot: How VLM responses degrade on imperfect chartscited by 5
  11. 2025Mitigating Diffusion Model Hallucinations with Dynamic Guidancecited by 5
  12. 2026Residual Decoding: Mitigating Hallucinations in Large Vision-Language Models via History-Aware Residual Guidancecited by 5
32 more
  1. 2025CCD: Mitigating Hallucinations in Radiology MLLMs via Clinical Contrastive Decodingcited by 4
  2. 2026OPGAgent: An Agent for Auditable Dental Panoramic X-ray Interpretationcited by 4
  3. 2025Seeing It or Not? Interpretable Vision-aware Latent Steering to Mitigate Object Hallucinationscited by 3
  4. 2025Reallocating Attention Across Layers to Reduce Multimodal Hallucinationcited by 3
  5. 2025HARMONY: Hidden Activation Representations and Model Output-Aware Uncertainty Estimation for Vision-Language Modelscited by 3
  6. 2025"It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with Vision-Language Modelscited by 3
  7. 2026WalkGPT: Grounded Vision-Language Conversation with Depth-Aware Segmentation for Pedestrian Navigationcited by 2
  8. 2026Aligning What Vision-Language Models See and Perceive with Adaptive Information Flowcited by 2
  9. 2025Adaptive Residual-Update Steering for Low-Overhead Hallucination Mitigation in Large Vision Language Modelscited by 1
  10. 2026Attention-space Contrastive Guidance for Efficient Hallucination Mitigation in LVLMscited by 1
  11. 2026MM-THEBench: Do Reasoning MLLMs Think Reasonably?cited by 1
  12. 2026SAKED: Mitigating Hallucination in Large Vision-Language Models via Stability-Aware Knowledge Enhanced Decodingcited by 1
  13. 2026GroundCount: Grounding Vision-Language Models with Object Detection for Mitigating Counting Hallucinationscited by 1
  14. 2026Seeing to Ground: Visual Attention for Hallucination-Resilient MDLLMscited by 1
  15. 2026Finding Distributed Object-Centric Properties in Self-Supervised Transformerscited by 1
  16. 2026Who Says What: Symbolic Trimodal Binding Mechanisms in Audio-Visual LLMscited by 1
  17. 2025Curing Semantic Drift: A Dynamic Approach to Grounding Generation in Large Vision-Language Modelscited by 0
  18. 2025Token-Level Inference-Time Alignment for Vision-Language Modelscited by 0
  19. 2025EraseLoRA: MLLM-Driven Foreground Exclusion and Background Subtype Aggregation for Dataset-Free Object Removalcited by 0
  20. 2026CPPO: Contrastive Perception Policy Optimization for VLM Agentscited by 0
  21. 2026Saliency-Aware Multi-Route Thinking: Revisiting Vision-Language Reasoningcited by 0
  22. 2026SaFeR-ToolKit: Structured Reasoning via Virtual Tool Calling for Multimodal Safetycited by 0
  23. 2026HTDC: Hesitation-Triggered Differential Calibration for Mitigating Hallucination in Large Vision-Language Modelscited by 0
  24. 2026Self-Captioning Multimodal Interaction Tuning: Amplifying Exploitable Redundancies for Robust Vision Language Modelscited by 0
  25. 2026Mitigating Object Hallucinations in Vision-Language Models through Region-Aware Attention Recalibrationcited by 0
  26. 2026What Makes LVLMs Hallucinate Less? Unveiling the Architectural Factors Behind Hallucination Robustnesscited by 0
  27. 2026PatchGate: Narrowing the Verbalization Gap with Intrinsic Object Inventories in Frozen Vision-Language Modelscited by 0
  28. 2026Fine-Grained Multi Image Object Hallucination Benchmarkcited by 0
  29. 2026DEEPO: Dual-Entropy Enhanced Policy Optimization for Hallucination in MLLMscited by 0
  30. 2026Beyond Saying Less: Fine-Grained Alignment for Informative and Faithful Vision-Language Modelscited by 0
  31. 2026CRAFT: Causal Responsibility and Failure Tracing in Medical Vision Language Modelscited by 0
  32. 2026Mitigating Object Hallucination in Large Vision-Language Models via False Discovery Controlled Visual Data Splittingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reducing Object Hallucination in LVLMs via Emphasizing Image-negative Tokens
  2. 2026Mitigating Hallucination in Vision-Language Models through Barrier-Regulated Adaptive Closed-form Steering
  3. 2026HIME: Mitigating Object Hallucinations in LVLMs via Hallucination Insensitivity Model Editing
  4. 2026Test-Time Hallucination Control in Large Vision-Language Models
  5. 2025Analyzing and Mitigating Object Hallucination: A Training Bias Perspective
  6. 2026Kestrel: Grounding Self-Refinement for LVLM Hallucination Mitigation
  7. 2024From Pixels to Tokens: Revisiting Object Hallucinations in Large Vision-Language Models
  8. 2026NoLan: Mitigating Object Hallucinations in Large Vision-Language Models via Dynamic Suppression of Language Priors