mlsanityData to 14 Oct 2026

Reducing Object Hallucination in LVLMs via Emphasizing Image-negative Tokens

arXiv 2605.21300 ↗First posted 20 May 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Object hallucination is a significant challenge that hinders the application of large vision-language models (LVLMs) in practice. We hypothesize that one possible origin of hallucination is the model's tendency to prioritize text generation over meaningful interaction with images. To explore this, we examine the generation process and categorize text tokens into three groups: image-positive, invariant, and negative, based on their visual dependence on input image tokens. Our analysis reveals that most generated tokens are minimally influenced by the image information. This suggests that during the model's training stage, more emphasis is placed on learning how to follow textual instructions, rather than extracting information from images. Based on this finding, we propose adjusting the training weights of different tokens depending on their visual dependence to control hallucination. Additionally, we remove a portion of the training data that potentially contains more hallucinations as a data filtering strategy. Both methods achieve a reduction in hallucination without compromising response length or introducing additional computational costs during inference. We validate our methods across three LVLM variants, demonstrating the effectiveness and general applicability.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 21 of its 25 references that are in the map. First posted on arXiv 20 May 2026.

Builds on

21 of 25 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Detecting and Evaluating Medical Hallucinations in Large Vision Language Models14 Jun
  2. 2024Multi-Modal Hallucination Control by Visual Information Grounding20 Mar
  3. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  4. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  5. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
  6. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
  7. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 14

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Visual Instruction Tuningcited by 7,869
  5. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  6. 2023VoxPoser: Composable 3D Value Maps for Robotic Manipulation with Language Modelscited by 750
  7. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 735
  8. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  9. 2024PaliGemma: A versatile 3B VLM for transfercited by 696
  10. 2018Object Hallucination in Image Captioningcited by 669
  11. 2023SVIT: Scaling up Visual Instruction Tuningcited by 162
  12. 2024Efficient Multimodal Learning from Data-centric Perspectivecited by 122
2 more
  1. 2023From CLIP to DINO: Visual Encoders Shout in Multi-modal Large Language Modelscited by 98
  2. 2022Visual Perturbation-aware Collaborative Learning for Overcoming the Language Prior Problemcited by 18

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Analyzing and Mitigating Object Hallucination: A Training Bias Perspective
  2. 2022Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-training
  3. 2025Enhancing Visual Reliance in Text Generation: A Bayesian Perspective on Mitigating Hallucination in Large Vision-Language Models
  4. 2024HELPD: Mitigating Hallucination of LVLMs by Hierarchical Feedback Learning with Vision-enhanced Penalty Decoding
  5. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steering
  6. 2026NoLan: Mitigating Object Hallucinations in Large Vision-Language Models via Dynamic Suppression of Language Priors
  7. 2026AdaIAT: Adaptively Increasing Attention to Generated Text to Alleviate Hallucinations in LVLM
  8. 2026CRoPS: A Training-Free Hallucination Mitigation Framework for Vision-Language Models