MHSA: A Lightweight Framework for Mitigating Hallucinations via Steered Attention in LVLMs
On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs
Abstract (arXiv)
Large vision-language models (LVLMs) have achieved remarkable performance across diverse multimodal tasks, yet they continue to suffer from hallucinations, generating content that is inconsistent with the visual input. Prior work DHCP (Detecting Hallucinations by Cross-modal Attention Pattern) has explored hallucination detection from the perspective of cross-modal attention, but does not address hallucination mitigation. In this paper, we propose MHSA (Mitigating Hallucinations via Steered Attention), a lightweight framework that mitigates hallucinations by learning to correct cross-modal attention patterns in LVLMs. MHSA trains a simple three-layer MLP generator to produce corrected attention, guided by supervisory signals from the DHCP discriminator and the LVLM itself. During inference, MHSA mitigates both discriminative and generative hallucinations across various datasets and LVLMs by simply replacing the original cross-modal attention with the corrected one, without modifying any LVLM parameters. By extending cross-modal attention mechanisms from hallucination detection to hallucination mitigation, MHSA offers a novel perspective on hallucination research in LVLMs and helps enhance their reliability.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 13 of its 31 references that are in the map. First posted on arXiv 14 May 2026.
Builds on
13 of 31 references.
Closest ancestors · 4
Same direction, newest first
- 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs31 Jul
- 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding27 Mar
- 2023Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning26 Jun
- 2023Evaluating Object Hallucination in Large Vision-Language Models17 May
Other references · 9
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
- 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
- 2020Learning to summarize from human feedbackcited by 2,562
- 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
- 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
- 2018Object Hallucination in Image Captioningcited by 661
- 2024PIP: Detecting Adversarial Examples in Large Vision-Language Models via Attention Patterns of Irrelevant Probe Questionscited by 14
- 2025Fighting Fire with Fire (F3): A Training-free and Efficient Visual Adversarial Example Purification Method in LVLMscited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025Not All Tokens and Heads Are Equally Important: Dual-Level Attention Intervention for Hallucination Mitigation
- 2024DHCP: Detecting Hallucinations by Cross-modal Attention Pattern in Large Vision-Language Models
- 2025VEGAS: Mitigating Hallucinations in Large Vision-Language Models via Vision-Encoder Attention Guided Adaptive Steering
- 2025MDSAM:Memory-Driven Sparse Attention Matrix for LVLMs Hallucination Mitigation
- 2024Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attention
- 2026See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs
- 2026Segmentation-Based Attention Entropy: Detecting and Mitigating Object Hallucinations in Large Vision-Language Models
- 2026Attention to details, logits to truth: visual-aware attention and logits enhancement to mitigate hallucinations in LVLMs