mlsanityData to 14 Oct 2026

Seeing Clearly by Layer Two: Enhancing Attention Heads to Alleviate Hallucination in LVLMs

arXiv 2411.09968 ↗First posted 15 Nov 2024cs.CV cs.AIcited by 52

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

The hallucination problem in multimodal large language models (MLLMs) remains a common issue. Although image tokens occupy a majority of the input sequence of MLLMs, there is limited research to explore the relationship between image tokens and hallucinations. In this paper, we analyze the distribution of attention scores for image tokens across each layer and head of the model, revealing an intriguing and common phenomenon: most hallucinations are closely linked to the pattern of attention sinks in the self-attention matrix of image tokens, where shallow layers exhibit dense attention sinks and deeper layers show sparse attention sinks. We further analyze the attention heads of different layers and find that heads with high-density attention sink in the image part play a positive role in alleviating hallucinations. In this paper, we propose a training-free method named textcolorredEnhancing textcolorredAttention textcolorredHeads (EAH), an approach designed to enhance the convergence of image tokens attention sinks in the shallow layers. EAH identifies the attention head that shows the vision sink in a shallow layer and extracts its attention matrix. This attention map is then broadcast to other heads in the layer, thereby strengthening the layer to pay more attention to the image itself. With extensive experiments, EAH shows significant hallucination-mitigating performance on different MLLMs and metrics, proving its effectiveness and generality.

Key facts. Cited by 52 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 37 of its 46 references that are in the map. First posted on arXiv 15 Nov 2024.

Builds on

37 of 46 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Mitigating Object Hallucination via Concentric Causal Attention21 Oct
  2. 2024Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attention18 Jun
  3. 2024Mitigating Dialogue Hallucination for Large Vision Language Models via Adversarial Instruction Tuning15 Mar
  4. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding1 Mar
  5. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Models23 Oct
  6. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
  7. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 30

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2023Mistral 7Bcited by 3,164
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  10. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  11. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  12. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
18 more
  1. 2024Qwen2 Technical Reportcited by 2,120
  2. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  3. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  4. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  5. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 940
  6. 2023A Survey on Multimodal Large Language Modelscited by 909
  7. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  8. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 872
  9. 2018Object Hallucination in Image Captioningcited by 669
  10. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 550
  11. 2023LLaMA-VID: An Image is Worth 2 Tokens in Large Language Modelscited by 546
  12. 2023DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Modelscited by 376
  13. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 365
  14. 2023MultiModal-GPT: A Vision and Language Model for Dialogue with Humanscited by 255
  15. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 236
  16. 2023Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructionscited by 77
  17. 2024Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibrationcited by 74
  18. 2024From Redundancy to Relevance: Information Flow in LVLMs Across Reasoning Taskscited by 38

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 21; 2026: 26.

52 papers.

Sort:

By area:

  1. 2025Kwai Keye-VL Technical Reportcited by 47
  2. 2025Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in Multimodal Large Language Modelscited by 22
  3. 2024Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergencecited by 19
  4. 2025DenseWorld-1M: Towards Detailed Dense Grounded Caption in the Real Worldcited by 16
  5. 2024Underwater Image Restoration via Polymorphic Large Kernel CNNscited by 14
  6. 2025Mitigating Hallucination in Large Vision-Language Models via Adaptive Attention Calibrationcited by 11
  7. 2026Attention Sinks Are Provably Necessary in Softmax Transformers: Evidence from Trigger-Conditional Taskscited by 11
  8. 2026Attention Sink in Transformers: A Survey on Utilization, Interpretation, and Mitigationcited by 11
  9. 2025Dense360: Dense Understanding from Omnidirectional Panoramascited by 9
  10. 2026Hallucination Begins Where Saliency Dropscited by 9
  11. 2025Focusing by Contrastive Attention: Enhancing VLMs' Visual Reasoningcited by 8
  12. 2025TARAC: Mitigating Hallucination in LVLMs via Temporal Attention Real-time Accumulative Connectioncited by 7
38 more
  1. 2025Revealing Perception and Generation Dynamics in LVLMs: Mitigating Hallucinations via Validated Dominance Correctioncited by 7
  2. 2024Attention with Dependency Parsing Augmentation for Fine-Grained Attributioncited by 5
  3. 2025AdaToken-3D: Dynamic Spatial Gating for Efficient 3D Large Multimodal-Models Reasoningcited by 5
  4. 2025Making Every Step Effective: Jailbreaking Large Vision-Language Models Through Hierarchical KV Equalizationcited by 4
  5. 2025A Comprehensive Analysis for Visual Object Hallucination in Large Vision-Language Modelscited by 4
  6. 2025Reallocating Attention Across Layers to Reduce Multimodal Hallucinationcited by 4
  7. 2026Revealing and Enhancing Core Visual Regions: Harnessing Internal Attention Dynamics for Hallucination Mitigation in LVLMscited by 4
  8. 2025METEOR: Multi-Encoder Collaborative Token Pruning for Efficient Vision Language Modelscited by 3
  9. 2025MAC-Lookup: Multi-Axis Conditional Lookup Model for Underwater Image Enhancementcited by 2
  10. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  11. 2025Cross-Layer Vision Smoothing: Enhancing Visual Understanding via Sustained Focus on Key Objects in Large Vision-Language Modelscited by 2
  12. 2026C^2ROPE: Causal Continuous Rotary Positional Encoding for 3D Large Multimodal-Models Reasoningcited by 2
  13. 2026AdaIAT: Adaptively Increasing Attention to Generated Text to Alleviate Hallucinations in LVLMcited by 2
  14. 2026SAGE: Sink-Aware Grounded Decoding for Multimodal Hallucination Mitigationcited by 2
  15. 2026Addressing Exacerbated Attention Sink for Source-Free Cross-Domain Few-Shot Learningcited by 2
  16. 2024Enhancing Multimodal Large Language Models Complex Reason via Similarity Computationcited by 1
  17. 2026Context-Aware Decoding for Faithful Vision-Language Generationcited by 1
  18. 2026Look Twice: Training-Free Evidence Highlighting for Knowledge-based Visual Question Answeringcited by 1
  19. 2026ART: Attention Replacement Technique to Improve Factuality in LLMscited by 1
  20. 2026When Looking Is Not Enough: Visual Attention Structure Reveals Hallucination in MLLMscited by 1
  21. 2026Risk-aware Selective Prompting for Hallucination Mitigation in Large Vision-Language Modelscited by 1
  22. 2026When Attention Collapses: Stage-Aware Visual Token Pruning from Structure to Semanticscited by 1
  23. 2026Clearer Sight, Fewer Lies: Oriented Pickup Preference Optimization for Multimodal Hallucination Mitigationcited by 1
  24. 2026ST-Veto: Spatio-Temporal Token Veto for Diffusion MLLMs via Taylor Prediction and Visual Groundingcited by 1
  25. 2024Toward Modality Gap: Vision Prototype Learning for Weakly-supervised Semantic Segmentation with CLIPcited by 0
  26. 2025First-place Solution for Streetscape Shop Sign Recognition Competitioncited by 0
  27. 2025Mitigating Object Hallucination via Robust Local Perception Searchcited by 0
  28. 2025SFormer: SNR-guided Transformer for Underwater Image Enhancement from the Frequency Domaincited by 0
  29. 2025A Comprehensive Analysis of Churn Prediction in Telecommunications Using Machine Learningcited by 0
  30. 2025Forestpest-YOLO: A High-Performance Detection Framework for Small Forestry Pestscited by 0
  31. 2026Scalpel: Fine-Grained Alignment of Attention Activation Manifolds via Mixture Gaussian Bridges to Mitigate Multimodal Hallucinationcited by 0
  32. 2026ACT Now: Preempting LVLM Hallucinations via Adaptive Context Integrationcited by 0
  33. 2026Uncertainty-Aware Exploratory Direct Preference Optimization for Multimodal Large Language Modelscited by 0
  34. 2026The Ebb and Flow of Multimodal Focus: Scheduling Visual Relay Windows for Grounded VLM Reasoningcited by 0
  35. 2026Role-Break in Attention Heads: Understanding and Detecting Hallucinations in VLMscited by 0
  36. 2026Evolving Safety Landscape of Multi-modal Large Language Models: A Survey of Emerging Threats and Safeguardscited by 0
  37. 2026Dynamic Alignment Compensation for Hallucination Mitigation in Large Vision-Language Modelscited by 0
  38. 2026Does Playing it Safe Count as Faithfulness? Reassessing LVLM Hallucination Mitigation Methodscited by 0

50 most cited of 52 All 52 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025PAINT: Paying Attention to INformed Tokens to Mitigate Hallucination in Large Vision-Language Model
  2. 2025Not All Tokens and Heads Are Equally Important: Dual-Level Attention Intervention for Hallucination Mitigation
  3. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens
  4. 2025MINT: Mitigating Hallucinations in Large Vision-Language Models via Token Reduction
  5. 2026Attention to details, logits to truth: visual-aware attention and logits enhancement to mitigate hallucinations in LVLMs
  6. 2025Mitigating Hallucination in Large Vision-Language Models through Aligning Attention Distribution to Information Flow
  7. 2025Causally-Grounded Dual-Path Attention Intervention for Object Hallucination Mitigation in LVLMs
  8. 2026Reducing Object Hallucination in LVLMs via Emphasizing Image-negative Tokens