mlsanityData to 14 Oct 2026

A Comprehensive Analysis for Visual Object Hallucination in Large Vision-Language Models

arXiv 2505.01958 ↗First posted 4 May 2025cs.CV cs.CLcited by 4

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Large Vision-Language Models (LVLMs) demonstrate remarkable capabilities in multimodal tasks, but visual object hallucination remains a persistent issue. It refers to scenarios where models generate inaccurate visual object-related information based on the query input, potentially leading to misinformation and concerns about safety and reliability. Previous works focus on the evaluation and mitigation of visual hallucinations, but the underlying causes have not been comprehensively investigated. In this paper, we analyze each component of LLaVA-like LVLMs -- the large language model, the vision backbone, and the projector -- to identify potential sources of error and their impact. Based on our observations, we propose methods to mitigate hallucination for each problematic component. Additionally, we developed two hallucination benchmarks: QA-VisualGenome, which emphasizes attribute and relation hallucinations, and QA-FB15k, which focuses on cognition-based hallucinations.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 57 references that are in the map. First posted on arXiv 4 May 2025.

Builds on

37 of 57 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024Seeing Clearly by Layer Two: Enhancing Attention Heads to Alleviate Hallucination in LVLMs15 Nov
  2. 2024Mitigating Object Hallucination via Concentric Causal Attention21 Oct
  3. 2024Investigating and Mitigating Object Hallucinations in Pretrained Vision-Language (CLIP) Models4 Oct
  4. 2024A Unified Hallucination Mitigation Framework for Large Vision-Language Models24 Sep
  5. 2024DOPRA: Decoding Over-accumulation Penalization and Re-allocation in Specific Weighting Layer21 Jul
  6. 2024Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attention18 Jun
  7. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding1 Mar
  8. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  9. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  10. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov
  11. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun

Other references · 26

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  7. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  9. 2015VQA: Visual Question Answeringcited by 2,506
  10. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  11. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  12. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
14 more
  1. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  2. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
  3. 2018nocaps: novel object captioning at scalecited by 444
  4. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  5. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 198
  6. 2023Silkie: Preference Distillation for Large Visual Language Modelscited by 130
  7. 2023M3IT: A Large-Scale Dataset towards Multi-Modal Multilingual Instruction Tuningcited by 126
  8. 2020Visual Commonsense R-CNNcited by 106
  9. 2021Conditional probing: measuring usable information beyond a baselinecited by 62
  10. 2024FGAIF: Aligning Large Vision-Language Models with Fine-grained AI Feedbackcited by 30
  11. 2024Deciphering Cross-Modal Alignment in Large Vision-Language Models with Modality Integration Ratecited by 20
  12. 2025Internal Activation Revision: Safeguarding Vision Language Models Without Parameter Updatecited by 16
  13. 2024Fine-grained and Explainable Factuality Evaluation for Multimodal Summarizationcited by 10
  14. 2024Defeasible Visual Entailment: Benchmark, Evaluator, and Reward-Driven Optimizationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2026Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?cited by 6
  2. 2025When to Think and When to Look: Uncertainty-Guided Lookbackcited by 3
  3. 2026Two Birds, One Projection: Harmonizing Safety and Utility in LVLMs via Inference-time Feature Projectioncited by 1
  4. 2026When Safety Overrides Vision: Exploring Dynamics between Vision Influence and Safety Alignment in Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Evaluating Object Hallucination in Large Vision-Language Models
  2. 2024From Pixels to Tokens: Revisiting Object Hallucinations in Large Vision-Language Models
  3. 2024A Survey on Hallucination in Large Vision-Language Models
  4. 2024Hallucination in Multimodal Foundation Models: A Survey on Causes, Corrections, and Evaluations
  5. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steering
  6. 2025Evaluating Hallucination in Large Vision-Language Models based on Context-Aware Object Similarities
  7. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models
  8. 2026NoLan: Mitigating Object Hallucinations in Large Vision-Language Models via Dynamic Suppression of Language Priors