mlsanityData to 14 Oct 2026

Countering the Over-Reliance Trap: Mitigating Object Hallucination for LVLMs via a Self-Validation Framework

arXiv 2601.22451 ↗First posted 30 Jan 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Despite progress in Large Vision Language Models (LVLMs), object hallucination remains a critical issue in image captioning task, where models generate descriptions of non-existent objects, compromising their reliability. Previous work attributes this to LVLMs' over-reliance on language priors and attempts to mitigate it through logits calibration. However, they still lack a thorough analysis of the over-reliance. To gain a deeper understanding of over-reliance, we conduct a series of preliminary experiments, indicating that as the generation length increases, LVLMs' over-reliance on language priors leads to inflated probability of hallucinated object tokens, consequently exacerbating object hallucination. To circumvent this issue, we propose Language-Prior-Free Verification to enable LVLMs to faithfully verify the confidence of object existence. Based on this, we propose a novel training-free Self-Validation Framework to counter the over-reliance trap. It first validates objects' existence in sampled candidate captions and further mitigates object hallucination via caption selection or aggregation. Experiment results demonstrate that our framework mitigates object hallucination significantly in image captioning task (e.g., 65.6% improvement on CHAIRI metric with LLaVA-v1.5-7B), surpassing the previous SOTA methods. This result highlights a novel path towards mitigating hallucination by unlocking the inherent potential within LVLMs themselves.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 16 of its 25 references that are in the map. First posted on arXiv 30 Jan 2026.

Builds on

16 of 25 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025Revealing Perception and Generation Dynamics in LVLMs: Mitigating Hallucinations via Validated Dominance Correction21 Dec
  2. 2025Med-VCD: Mitigating Hallucination for Medical Large Vision Language Models through Visual Contrastive Decoding1 Dec
  3. 2025CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs28 Jan
  4. 2024Nullu: Mitigating Object Hallucinations in Large Vision-Language Models via HalluSpace Projection18 Dec
  5. 2024MLLM can see? Dynamic Correction Decoding for Hallucination Mitigation15 Oct
  6. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models4 Aug
  7. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs31 Jul
  8. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  9. 2023HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Data22 Nov
  10. 2023Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Models9 Oct
  11. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun

Other references · 5

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  3. 2018Object Hallucination in Image Captioningcited by 669
  4. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  5. 2020Counterfactual VQA: A Cause-Effect Look at Language Biascited by 266

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023HallE-Control: Controlling Object Hallucination in Large Multimodal Models
  2. 2024Do More Details Always Introduce More Hallucinations in LVLM-based Image Captioning?
  3. 2023Evaluating Object Hallucination in Large Vision-Language Models
  4. 2026From Token Correction to Claim Admission: Training-Free Hallucination Mitigation for Large Vision-Language Models
  5. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models
  6. 2023Mitigating Fine-Grained Hallucination by Fine-Tuning Large Vision-Language Models with Caption Rewrites
  7. 2024Does Object Grounding Really Reduce Hallucination of Large Vision-Language Models?
  8. 2026TruthLens: Object Hallucination Detection via Self-Evaluating Truthfulness Scores in LVLMs