mlsanityData to 14 Oct 2026

Investigating and Mitigating Object Hallucinations in Pretrained Vision-Language (CLIP) Models

arXiv 2410.03176 ↗First posted 4 Oct 2024cs.CV cs.AIcited by 15

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Large Vision-Language Models (LVLMs) have achieved impressive performance, yet research has pointed out a serious issue with object hallucinations within these models. However, there is no clear conclusion as to which part of the model these hallucinations originate from. In this paper, we present an in-depth investigation into the object hallucination problem specifically within the CLIP model, which serves as the backbone for many state-of-the-art vision-language systems. We unveil that even in isolation, the CLIP model is prone to object hallucinations, suggesting that the hallucination problem is not solely due to the interaction between vision and language modalities. To address this, we propose a counterfactual data augmentation method by creating negative samples with a variety of hallucination issues. We demonstrate that our method can effectively mitigate object hallucinations for CLIP model, and we show the the enhanced model can be employed as a visual encoder, effectively alleviating the object hallucination issue in LVLMs.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 56 references that are in the map. First posted on arXiv 4 Oct 2024.

Builds on

36 of 56 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  2. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization28 Nov
  3. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov
  4. 2023CIEM: Contrastive Instruction Evaluation Method for Better Instruction Tuning5 Sep

Other references · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  5. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  6. 2020Learning to summarize from human feedbackcited by 2,588
  7. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  8. 2015VQA: Visual Question Answeringcited by 2,506
  9. 2013Describing Textures in the Wildcited by 2,125
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  11. 2019Towards VQA Models That Can Readcited by 1,718
  12. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
20 more
  1. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  2. 2022No Language Left Behind: Scaling Human-Centered Machine Translationcited by 1,312
  3. 2016Modeling Context in Referring Expressionscited by 1,063
  4. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  5. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  6. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  7. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  8. 2018nocaps: novel object captioning at scalecited by 444
  9. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 360
  10. 2023Demystifying CLIP Datacited by 270
  11. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 256
  12. 2023What's "up" with vision-language models? Investigating their struggle with spatial reasoningcited by 254
  13. 2023Data Filtering Networkscited by 250
  14. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 243
  15. 2024Aligning Modalities in Vision Large Language Models via Preference Fine-tuningcited by 198
  16. 2023To See is to Believe: Prompting GPT-4V for Better Visual Instruction Tuningcited by 144
  17. 2017FOIL it! Find One mismatch between Image and Language captioncited by 95
  18. 2021Let there be a clock on the beach: Reducing Object Hallucination in Image Captioningcited by 76
  19. 2023CLIPA-v2: Scaling CLIP Training with 81.1% Zero-shot ImageNet Accuracy within a 10,000 Budget; An Extra 4,000 Unlocks 81.8% Accuracycited by 24
  20. 2023NLLB-CLIP -- train performant multilingual image retrieval model on a budgetcited by 24

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 9; 2026: 6.

15 papers.

Sort:

By area:

  1. 2025SteerVLM: Robust Model Control through Lightweight Activation Steering for Vision Language Modelscited by 8
  2. 2025Enhancing Concept Localization in CLIP-based Concept Bottleneck Modelscited by 6
  3. 2025Image Tokens Matter: Mitigating Hallucination in Discrete Tokenizer-based Large Vision-Language Models via Latent Editingcited by 5
  4. 2025A Comprehensive Analysis for Visual Object Hallucination in Large Vision-Language Modelscited by 4
  5. 2026Beyond Linear Activation Steering: Invertible Latent Transformations for Controlling LLM Behaviorcited by 3
  6. 2025Do Vision Encoders Truly Explain Object Hallucination?: Mitigating Object Hallucination via Simple Fine-Grained CLIPScorecited by 2
  7. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  8. 2025Controlling Multimodal LLMs via Reward-guided Decodingcited by 2
  9. 2026When Prompts Override Vision: Prompt-Induced Hallucinations in LVLMscited by 2
  10. 2025Diving into Mitigating Hallucinations from a Vision Perspective for Large Vision-Language Modelscited by 1
  11. 2026Disentangling Hallucinations: Orthogonal Semantic Projection for Robust Interpretabilitycited by 1
  12. 2025Look Closer! An Adversarial Parametric Editing Framework for Hallucination Mitigation in VLMscited by 0
3 more
  1. 2026Through the Lens of Character: Resolving Modality-Role Interference in Multimodal Role-Playing Agentcited by 0
  2. 2026Rethinking CD: A Reproducibility Study and Extension on the Ineffectiveness of Contrastive Decoding at Mitigating Object Hallucinations in MLLMscited by 0
  3. 2026DARE to Mitigate Hallucination: Dual-path Auto-Regressive-aware Editingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026NoLan: Mitigating Object Hallucinations in Large Vision-Language Models via Dynamic Suppression of Language Priors
  2. 2024Seeing is Believing: Mitigating Hallucination in Large Vision-Language Models via CLIP-Guided Decoding
  3. 2024From Pixels to Tokens: Revisiting Object Hallucinations in Large Vision-Language Models
  4. 2023Evaluating Object Hallucination in Large Vision-Language Models
  5. 2024A Survey on Hallucination in Large Vision-Language Models
  6. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steering
  7. 2025Analyzing and Mitigating Object Hallucination: A Training Bias Perspective
  8. 2025Watch Closely: Mitigating Object Hallucinations in Large Vision-Language Models with Disentangled Decoding