mlsanityData to 14 Oct 2026

Visual Attention Drifts,but Anchors Hold:Mitigating Hallucination in Multimodal Large Language Models via Cross-Layer Visual Anchors

arXiv 2603.25088 ↗First posted 26 Mar 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Multimodal Large Language Models often suffer from object hallucination. While existing research utilizes attention enhancement and visual retracing, we find these works lack sufficient interpretability regarding attention drift in final model stages. In this paper, we investigate the layer wise evolution of visual features and discover that hallucination stems from deep layer attention regressing toward initial visual noise from early layers. We observe that output reliability depends on acquiring visual anchors at intermediate layers rather than final layers. Based on these insights, we propose CLVA, which stands for Cross-Layer Visual Anchors, a training free method that reinforces critical mid layer features while suppressing regressive noise. This approach effectively pulls deep layer attention back to correct visual regions by utilizing essential anchors captured from attention dynamics. We evaluate our method across diverse architectures and benchmarks, demonstrating outstanding performance without significant increase in computational time and GPU memory.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 33 of its 38 references that are in the map. First posted on arXiv 26 Mar 2026.

Builds on

33 of 38 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025ASCD: Attention-Steerable Contrastive Decoding for Reducing Hallucination in MLLM17 Jun
  2. 2025Not All Tokens and Heads Are Equally Important: Dual-Level Attention Intervention for Hallucination Mitigation14 Jun
  3. 2025Do You Keep an Eye on What I Ask? Mitigating Multimodal Hallucination via Attention-Guided Ensemble Decoding23 May
  4. 2025ClearSight: Visual Signal Enhancement for Object Hallucination Mitigation in Multimodal Large language Models17 Mar
  5. 2025The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models via Visual Information Steering5 Feb
  6. 2025Cross-Modal Attention Calibration for LVLM Hallucination Mitigation3 Jan
  7. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens23 Nov
  8. 2024MLLM can see? Dynamic Correction Decoding for Hallucination Mitigation15 Oct
  9. 2024Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Models4 Oct
  10. 2024Hallucination of Multimodal Large Language Models: A Survey29 Apr
  11. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding27 Mar
  12. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
2 more
  1. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  2. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 19

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  3. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,322
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  5. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  7. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  8. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  9. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  10. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  11. 2018Object Hallucination in Image Captioningcited by 669
  12. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 482
7 more
  1. 2025MLLMs Know Where to Look: Training-free Perception of Small Visual Details with Multimodal LLMscited by 176
  2. 2024RLAIF-V: Open-Source AI Feedback Leads to Super GPT-4V Trustworthinesscited by 131
  3. 2025Token Activation Map to Visually Explain Multimodal LLMscited by 32
  4. 2024Exploring Perceptual Limitation of Multimodal Large Language Modelscited by 25
  5. 2025POPEN: Preference-Based Optimization and Ensemble for LVLM-Based Reasoning Segmentationcited by 22
  6. 2024The Evolution of Multimodal Model Architecturescited by 21
  7. 2025Heaven-Sent or Hell-Bent? Benchmarking the Intelligence and Defectiveness of LLM Hallucinationscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026ADAPT: Attention Dynamics Alignment with Preference Tuning for Faithful MLLMs
  2. 2025CLAIM: Mitigating Multilingual Object Hallucination in Large Vision-Language Models with Cross-Lingual Attention Intervention
  3. 2026Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation
  4. 2025Extracting Visual Facts from Intermediate Layers for Mitigating Hallucinations in Multimodal Large Language Models
  5. 2026Self-Correction Inside the Model: Leveraging Layer Attention to Mitigate Hallucinations in Large Vision Language Models
  6. 2024Seeing Clearly by Layer Two: Enhancing Attention Heads to Alleviate Hallucination in LVLMs
  7. 2025CAI: Caption-Sensitive Attention Intervention for Mitigating Object Hallucination in Large Vision-Language Models
  8. 2024Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence