mlsanityData to 7 Oct 2026

Instruction-Aligned Visual Attention for Mitigating Hallucinations in Large Vision-Language Models

arXiv 2503.18556 ↗First posted 24 Mar 2025cs.CV cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

Despite the significant success of Large Vision-Language models(LVLMs), these models still suffer hallucinations when describing images, generating answers that include non-existent objects. It is reported that these models tend to over-focus on certain irrelevant image tokens that do not contain critical information for answering the question and distort the output. To address this, we propose an Instruction-Aligned Visual Attention(IAVA) approach, which identifies irrelevant tokens by comparing changes in attention weights under two different instructions. By applying contrastive decoding, we dynamically adjust the logits generated from original image tokens and irrelevant image tokens, reducing the model's over-attention to irrelevant information. The experimental results demonstrate that IAVA consistently outperforms existing decoding techniques on benchmarks such as MME, POPE, and TextVQA in mitigating object hallucinations. Our IAVA approach is available online at https://github.com/Lee-lab558/IAVA.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 38 of its 49 references that are in the map. First posted on arXiv 24 Mar 2025.

Builds on

38 of 49 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025LanP: Rethinking the Impact of Language Priors in Large Vision-Language Models17 Feb
  2. 2024DAMRO: Dive into the Attention Mechanism of LVLM to Reduce Object Hallucination6 Oct
  3. 2024Do More Details Always Introduce More Hallucinations in LVLM-based Image Captioning?18 Jun
  4. 2024Don't Miss the Forest for the Trees: Attentional Vision Calibration for Large Vision Language Models28 May
  5. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding27 Mar
  6. 2024Multi-Modal Hallucination Control by Visual Information Grounding20 Mar
  7. 2024A Survey on Hallucination in Large Vision-Language Models1 Feb
  8. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  9. 2023Woodpecker: Hallucination Correction for Multimodal Large Language Models24 Oct
  10. 2023CIEM: Contrastive Instruction Evaluation Method for Better Instruction Tuning5 Sep

Other references · 28

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  7. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  8. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  9. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  10. 2019A Survey of Autonomous Driving: Common Practices and Emerging Technologiescited by 373
  11. 2021Attention Mechanisms in Computer Vision: A Surveycited by 232
  12. 2023What does CLIP know about a red circle? Visual prompt engineering for VLMscited by 215
16 more
  1. 2024InternLM-XComposer2-4KHD: A Pioneering Large Vision-Language Model Handling Resolutions from 336 Pixels to 4K HDcited by 156
  2. 2016Visual Question Answering: A Survey of Methods and Datasetscited by 123
  3. 2023Talk2BEV: Language-enhanced Bird's-eye View Maps for Autonomous Drivingcited by 88
  4. 2021Kaleido-BERT: Vision-Language Pre-training on Fashion Domaincited by 67
  5. 2020FashionBERT: Text and Image Matching with Adaptive Loss for Cross-modal Retrievalcited by 55
  6. 2024Looking Beyond Text: Reducing Language bias in Large Vision-Language Models via Multimodal Dual-Attention and Soft-Image Guidancecited by 32
  7. 2023MLLMs-Augmented Visual-Language Representation Learningcited by 21
  8. 2023Guiding Image Captioning Models Toward More Specific Captionscited by 17
  9. 2024HiRes-LLaVA: Restoring Fragmentation Input in High-Resolution Large Vision-Language Modelscited by 16
  10. 2022Masked Vision-Language Transformer in Fashioncited by 14
  11. 2024Leveraging Large Vision-Language Model as User Intent-aware Encoder for Composed Image Retrievalcited by 13
  12. 2023VQA4CIR: Boosting Composed Image Retrieval with Visual Question Answeringcited by 11
  13. 2024Hallucination in Multimodal Foundation Models: A Survey on Causes, Corrections, and Evaluationscited by 8
  14. 2024General2Specialized LLMs Translation for E-commercecited by 6
  15. 2024MoDULA: Mixture of Domain-Specific and Universal LoRA for Multi-Task Learningcited by 5
  16. 2024CoF: Coarse to Fine-Grained Image Understanding for Multi-modal Large Language Modelscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Attention to details, logits to truth: visual-aware attention and logits enhancement to mitigate hallucinations in LVLMs
  2. 2025Mixture of Decoding: An Attention-Inspired Adaptive Decoding Strategy to Mitigate Hallucinations in Large Vision-Language Models
  3. 2026See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs
  4. 2025Mitigating Hallucinations in Vision-Language Models through Image-Guided Head Suppression
  5. 2025IKOD: Mitigating Visual Attention Degradation in Large Vision-Language Models
  6. 2025MINT: Mitigating Hallucinations in Large Vision-Language Models via Token Reduction
  7. 2025Mitigating Hallucination in Large Vision-Language Models through Aligning Attention Distribution to Information Flow
  8. 2025VEGAS: Mitigating Hallucinations in Large Vision-Language Models via Vision-Encoder Attention Guided Adaptive Steering