mlsanityData to 14 Oct 2026

Mitigating Hallucinations in Vision-Language Models through Image-Guided Head Suppression

arXiv 2505.16411 ↗First posted 22 May 2025cs.CV cs.LGcited by 21

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Despite their remarkable progress in multimodal understanding tasks, large vision language models (LVLMs) often suffer from "hallucinations", generating texts misaligned with the visual context. Existing methods aimed at reducing hallucinations through inference time intervention incur a significant increase in latency. To mitigate this, we present SPIN, a task-agnostic attention-guided head suppression strategy that can be seamlessly integrated during inference, without incurring any significant compute or latency overhead. We investigate whether hallucination in LVLMs can be linked to specific model components. Our analysis suggests that hallucinations can be attributed to a dynamic subset of attention heads in each layer. Leveraging this insight, for each text query token, we selectively suppress attention heads that exhibit low attention to image tokens, keeping the top-K attention heads intact. Extensive evaluations on visual question answering and image description tasks demonstrate the efficacy of SPIN in reducing hallucination scores up to 2.7x while maintaining F1, and improving throughput by 1.8x compared to existing alternatives. Code is available at https://github.com/YUECHE77/SPIN.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 33 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

26 of 33 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024ICT: Image-Object Cross-Level Trusted Intervention for Mitigating Object Hallucination in Large Vision-Language Models22 Nov
  2. 2024DAMRO: Dive into the Attention Mechanism of LVLM to Reduce Object Hallucination6 Oct
  3. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs31 Jul
  4. 2024HALC: Object Hallucination Reduction via Adaptive Focal-Contrast Decoding1 Mar
  5. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  6. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  7. 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun

Other references · 19

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2025Qwen2.5-VL Technical Reportcited by 5,587
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  5. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  6. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  7. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  8. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  9. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 906
  10. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  11. 2023Large Language Modelscited by 780
  12. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
7 more
  1. 2017Beam Search Strategies for Neural Machine Translationcited by 232
  2. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 158
  3. 2024MoH: Multi-Head Attention as Mixture-of-Head Attentioncited by 44
  4. 2024Looking Beyond Text: Reducing Language bias in Large Vision-Language Models via Multimodal Dual-Attention and Soft-Image Guidancecited by 32
  5. 2024ShiftAddLLM: Accelerating Pretrained LLMs via Post-Training Multiplication-Less Reparameterizationcited by 30
  6. 2025CITER: Collaborative Inference for Efficient Large Language Model Decoding with Token-Level Routingcited by 30
  7. 2025LVLM-Compress-Bench: Benchmarking the Broader Impact of Large Vision-Language Model Compressioncited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 13.

21 papers.

Sort:

By area:

  1. 2025When Seeing Overrides Knowing: Disentangling Knowledge Conflicts in Vision-Language Modelscited by 11
  2. 2025Tracing and Mitigating Hallucinations in Multimodal LLMs via Dynamic Attention Localizationcited by 4
  3. 2025"It's trained by non-disabled people": Evaluating How Image Quality Affects Product Captioning with Vision-Language Modelscited by 3
  4. 2025When to Think and When to Look: Uncertainty-Guided Lookbackcited by 3
  5. 2025VEGAS: Mitigating Hallucinations in Large Vision-Language Models via Vision-Encoder Attention Guided Adaptive Steeringcited by 3
  6. 2026Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gatingcited by 2
  7. 2026ADAPT: Attention Dynamics Alignment with Preference Tuning for Faithful MLLMscited by 2
  8. 2026SeeMe: Mitigating Hallucinations in Large Vision-Language Models through Effective Visual Token Engineeringcited by 2
  9. 2025MAP: Mitigating Hallucinations in Large Vision-Language Models with Map-Level Attention Processingcited by 1
  10. 2025PAS : Prelim Attention Score for Detecting Object Hallucinations in Large Vision--Language Modelscited by 1
  11. 2026When RAG Hurts: Diagnosing and Mitigating Attention Distraction in Retrieval-Augmented LVLMscited by 1
  12. 2026Seeing to Ground: Visual Attention for Hallucination-Resilient MDLLMscited by 1
9 more
  1. 2026GEASS: Gated Evidence-Adaptive Selective Caption Trust for Vision-Language Modelscited by 1
  2. 2026Targeting the Attention Heads Behind Object Hallucination in LLaVAcited by 1
  3. 2025Conscious Gaze: Adaptive Attention Mechanisms for Hallucination Mitigation in Vision-Language Modelscited by 0
  4. 2026System-Mediated Attention Imbalances Make Vision-Language Models Say Yescited by 0
  5. 2026Logit Lens Supervision for Patch-Level Explanations in Vision-Language Modelscited by 0
  6. 2026Mitigating Object Hallucinations in Vision-Language Models through Region-Aware Attention Recalibrationcited by 0
  7. 2026Mitigating Hallucination in Vision-Language Models through Barrier-Regulated Adaptive Closed-form Steeringcited by 0
  8. 2026Learning from Fine-Grained Visual Discrepancies: Mitigating Multimodal Hallucinations via In-Context Visual Contrastive Optimizationcited by 0
  9. 2026Role-Break in Attention Heads: Understanding and Detecting Hallucinations in VLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Focus Matters: Phase-Aware Suppression for Hallucination in Vision-Language Models
  2. 2025Not All Tokens and Heads Are Equally Important: Dual-Level Attention Intervention for Hallucination Mitigation
  3. 2024A Unified Hallucination Mitigation Framework for Large Vision-Language Models
  4. 2026See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs
  5. 2025MINT: Mitigating Hallucinations in Large Vision-Language Models via Token Reduction
  6. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steering
  7. 2025Mixture of Decoding: An Attention-Inspired Adaptive Decoding Strategy to Mitigate Hallucinations in Large Vision-Language Models
  8. 2024Mitigating Hallucinations in Large Vision-Language Models via Summary-Guided Decoding