mlsanityData to 14 Oct 2026

Capturing Gaze Shifts for Guidance: Cross-Modal Fusion Enhancement for VLM Hallucination Mitigation

arXiv 2510.22067 ↗First posted 24 Oct 2025cs.CVcited by 5

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Vision language models (VLMs) often generate hallucination, i.e., content that cannot be substantiated by either textual or visual inputs. Prior work primarily attributes this to over-reliance on linguistic prior knowledge rather than visual inputs. Some methods attempt to mitigate hallucination by amplifying visual token attention proportionally to their attention scores. However, these methods overlook the visual attention sink problem, where attention is frequently misallocated to task-irrelevant visual regions, and neglect cross-modal fusion balance by enhancing only visual attention without adjusting attention to the user query. This can result in amplifying incorrect areas while failing to properly interpret the user query. To address these challenges, we propose a simple yet effective method called Gaze Shift-Guided Cross-modal Fusion Enhancement (GIFT). GIFT pre-computes a holistic visual saliency map by tracking positive changes in visual attention, or "gaze shifts", during user query comprehension, and leverages this map to amplify attention to both salient visual information and the user query at each decoding step. This reduces the impact of visual attention sink, as irrelevant tokens exhibit minimal shifts, while ensuring balanced cross-modal fusion for well-integrated representation. Extensive experiments show that GIFT effectively mitigates hallucination in VLMs across both generative and classification tasks, achieving up to 20.7% improvement over greedy decoding, while maintaining general vision-language performance with low computational overhead.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 50 of its 56 references that are in the map. First posted on arXiv 24 Oct 2025.

Builds on

50 of 56 references.

Closest ancestors · 19

Same direction, newest first

  1. 2025Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resampling17 Apr
  2. 2025TARAC: Mitigating Hallucination in LVLMs via Temporal Attention Real-time Accumulative Connection5 Apr
  3. 2025ClearSight: Visual Signal Enhancement for Object Hallucination Mitigation in Multimodal Large language Models17 Mar
  4. 2025Through the Magnifying Glass: Adaptive Perception Magnification for Hallucination-Free VLM Decoding13 Mar
  5. 2025PerturboLLaVA: Reducing Multimodal Hallucinations with Perturbative Visual Training9 Mar
  6. 2025Mitigating Object Hallucinations in Large Vision-Language Models via Attention Calibration4 Feb
  7. 2025MINT: Mitigating Hallucinations in Large Vision-Language Models via Token Reduction2 Feb
  8. 2024Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence18 Dec
  9. 2024Mitigating Hallucinations in Large Vision-Language Models via Summary-Guided Decoding17 Oct
  10. 2024MLLM can see? Dynamic Correction Decoding for Hallucination Mitigation15 Oct
  11. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models4 Aug
  12. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs31 Jul
7 more
  1. 2024Mitigating Object Hallucination in MLLMs via Data-augmented Phrase-level Alignment28 May
  2. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decoding28 Feb
  3. 2024A Survey on Hallucination in Large Vision-Language Models1 Feb
  4. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  5. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov
  6. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
  7. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 31

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  4. 2025Qwen2.5-VL Technical Reportcited by 5,587
  5. 2024GPT-4o System Cardcited by 4,588
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  7. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  8. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  9. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  10. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
  11. 2019Towards VQA Models That Can Readcited by 1,718
  12. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
19 more
  1. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  2. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,305
  3. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  4. 2023Vision Transformers Need Registerscited by 871
  5. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  6. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  7. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 706
  8. 2018Object Hallucination in Image Captioningcited by 669
  9. 2025Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Freecited by 283
  10. 2023DriveMLM: Aligning Multi-Modal Large Language Models with Behavioral Planning States for Autonomous Drivingcited by 233
  11. 2024HuatuoGPT-Vision, Towards Injecting Medical Visual Knowledge into Multimodal LLMs at Scalecited by 200
  12. 2025MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMscited by 176
  13. 2025See What You Are Told: Visual Attention Sink in Large Multimodal Modelscited by 132
  14. 2024Information Flow Routes: Automatically Interpreting Language Models at Scalecited by 95
  15. 2023Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMscited by 91
  16. 2024Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimizationcited by 75
  17. 2024LLaRA: Supercharging Robot Learning Data for Vision-Language Policycited by 52
  18. 2024Looking Beyond Text: Reducing Language bias in Large Vision-Language Models via Multimodal Dual-Attention and Soft-Image Guidancecited by 32
  19. 2024Attention Prompting on Image for Large Vision-Language Modelscited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Attention Sink in Transformers: A Survey on Utilization, Interpretation, and Mitigationcited by 11
  2. 2026Revealing and Enhancing Core Visual Regions: Harnessing Internal Attention Dynamics for Hallucination Mitigation in LVLMscited by 4
  3. 2026SAGE: Sink-Aware Grounded Decoding for Multimodal Hallucination Mitigationcited by 2
  4. 2026Beyond Static Cropping: Layer-Adaptive Visual Localization and Decoding Enhancementcited by 1
  5. 2026GesBench: When and Where Do Multimodal Language Models Perceive Social Meaning?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025When Images Speak Louder: Mitigating Language Bias-induced Hallucinations in VLMs through Cross-Modal Guidance
  2. 2025Not All Tokens and Heads Are Equally Important: Dual-Level Attention Intervention for Hallucination Mitigation
  3. 2025Cross-Modal Attention Calibration for LVLM Hallucination Mitigation
  4. 2025Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs
  5. 2026Beyond Visual Enhancement: Adaptive Multi-Context Steering to Mitigate LVLM Hallucinations
  6. 2026Finding the Correct Visual Evidence Without Forgetting: Mitigating Hallucination in LVLMs via Inter-Layer Visual Attention Discrepancy
  7. 2026Attention to details, logits to truth: visual-aware attention and logits enhancement to mitigate hallucinations in LVLMs
  8. 2025Modality Bias in LVLMs: Analyzing and Mitigating Object Hallucination via Attention Lens