mlsanityData to 14 Oct 2026

Do Vision Encoders Truly Explain Object Hallucination?: Mitigating Object Hallucination via Simple Fine-Grained CLIPScore

arXiv 2502.20034 ↗First posted 27 Feb 2025cs.CV cs.CLcited by 2

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Recently, Large Vision-Language Models (LVLMs) show remarkable performance across various domains. However, these models suffer from object hallucination. In this work, we study object hallucination primarily in a discriminative, retrieval-style evaluation setting (OHD-Caps), rather than in free-form caption generation. This study revisits the previous claim that the cause of such hallucinations lies in the limited representational capacity of the vision encoder. Our analysis implies that the capacity of the vision encoder is not necessarily a major limiting factor in detecting object hallucination. Based on this insight, we propose Fine-grained CLIPScore (F-CLIPScore), a simple yet effective evaluation metric that enhances object-level granularity by incorporating text embeddings at the noun level. Evaluations on the OHD-Caps benchmark show that F-CLIPScore significantly outperforms conventional CLIPScore in accuracy by a large margin of 39.6% without additional training. We further demonstrate that F-CLIPScore-based data filtering reduces object hallucination in LVLM (4.9% in POPE accuracy after alignment pretraining). Our code is publicly available at https://github.com/abzb1/f-clip

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 23 of its 25 references that are in the map. First posted on arXiv 27 Feb 2025.

Builds on

23 of 25 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Investigating and Mitigating Object Hallucinations in Pretrained Vision-Language (CLIP) Models4 Oct
  2. 2024Mitigating Hallucinations in Large Vision-Language Models (LVLMs) via Language-Contrastive Decoding (LCD)6 Aug
  3. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding27 Mar
  4. 2024A Survey on Hallucination in Large Vision-Language Models1 Feb
  5. 2023Hallucination Augmented Contrastive Learning for Multimodal Large Language Model12 Dec
  6. 2023CIEM: Contrastive Instruction Evaluation Method for Better Instruction Tuning5 Sep
  7. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
  8. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 15

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  4. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  7. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,424
  8. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  9. 2021LAION-400M: Open Dataset of CLIP-Filtered 400 Million Image-Text Pairscited by 1,555
  10. 2023EVA-CLIP: Improved Training Techniques for CLIP at Scalecited by 798
  11. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 698
  12. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
3 more
  1. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  2. 2024Scaling Laws for Data Filtering -- Data Curation cannot be Compute Agnosticcited by 77
  3. 2023Contrasting Intra-Modal and Ranking Cross-Modal Hard Negatives to Enhance Visio-Linguistic Compositional Understandingcited by 41

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  2. 2026When More Words Say Less: Decoupling Length and Specificity in Image Description Evaluationcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2024Seeing is Believing: Mitigating Hallucination in Large Vision-Language Models via CLIP-Guided Decoding
  2. 2025Diving into Mitigating Hallucinations from a Vision Perspective for Large Vision-Language Models
  3. 2022Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-training
  4. 2026NoLan: Mitigating Object Hallucinations in Large Vision-Language Models via Dynamic Suppression of Language Priors
  5. 2025Evaluating Hallucination in Large Vision-Language Models based on Context-Aware Object Similarities
  6. 2025Exploring Causes and Mitigation of Hallucinations in Large Vision Language Models
  7. 2025Analyzing and Mitigating Object Hallucination: A Training Bias Perspective
  8. 2023Mitigating Fine-Grained Hallucination by Fine-Tuning Large Vision-Language Models with Caption Rewrites