mlsanityData to 14 Oct 2026

Exploring Causes and Mitigation of Hallucinations in Large Vision Language Models

arXiv 2502.16842 ↗First posted 24 Feb 2025cs.CVcited by 2

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Large Vision-Language Models (LVLMs) integrate image encoders with Large Language Models (LLMs) to process multi-modal inputs and perform complex visual tasks. However, they often generate hallucinations by describing non-existent objects or attributes, compromising their reliability. This study analyzes hallucination patterns in image captioning, showing that not all tokens in the generation process are influenced by image input and that image dependency can serve as a useful signal for hallucination detection. To address this, we develop an automated pipeline to identify hallucinated objects and train a token-level classifier using hidden representations from parallel inference passes-with and without image input. Leveraging this classifier, we introduce a decoding strategy that effectively controls hallucination rates in image captioning at inference time.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 28 references that are in the map. First posted on arXiv 24 Feb 2025.

Builds on

26 of 28 references.

Closest ancestors · 10

Same direction, newest first

  1. 2024Multi-Object Hallucination in Vision-Language Models8 Jul
  2. 2024Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Models30 Jun
  3. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  4. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  5. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization28 Nov
  6. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov
  7. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Models24 Oct
  8. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
  9. 2023Detecting and Preventing Hallucinations in Large Vision Language Models11 Aug
  10. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 16

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  8. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  10. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  11. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  12. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
4 more
  1. 2018Object Hallucination in Image Captioningcited by 669
  2. 2024YOLO-World: Real-Time Open-Vocabulary Object Detectioncited by 503
  3. 2024Why are Visually-Grounded Language Models Bad at Image Classification?cited by 102
  4. 2023TagCLIP: A Local-to-Global Framework to Enhance Open-Vocabulary Multi-Label Classification of CLIP Without Trainingcited by 19

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026SAVAA: Mitigating Hallucinations in LVLMs via Step-wise Adaptive Visual Attention Amplificationcited by 2
  2. 2026Improving Medical Visual Reinforcement Fine-Tuning via Perception and Reasoning Augmentationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steering
  2. 2024Detecting and Mitigating Hallucination in Large Vision Language Models via Fine-Grained AI Feedback
  3. 2025Watch Closely: Mitigating Object Hallucinations in Large Vision-Language Models with Disentangled Decoding
  4. 2024From Pixels to Tokens: Revisiting Object Hallucinations in Large Vision-Language Models
  5. 2026Reducing Object Hallucination in LVLMs via Emphasizing Image-negative Tokens
  6. 2026NoLan: Mitigating Object Hallucinations in Large Vision-Language Models via Dynamic Suppression of Language Priors
  7. 2025Diving into Mitigating Hallucinations from a Vision Perspective for Large Vision-Language Models
  8. 2026HALP: Detecting Hallucinations in Vision-Language Models without Generating a Single Token