# Mitigating Hallucination in Large Vision-Language Models through Aligning Attention Distribution to Information Flow

arXiv: https://arxiv.org/abs/2505.14257 · HTML: https://mlsanity.com/paper/2505.14257/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 9 of its 43 references that are in the map.
- First posted on arXiv 20 May 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Hallucination mitigation in LVLMs](https://mlsanity.com/mllm/hallucination-mitigation-in-lvlms/)

## Builds on (most cited first, 9 of 9)

- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [A Survey on Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2402.00253/) (2024)
- [AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation](https://mlsanity.com/paper/2311.07397/) (2023)
- [Evaluation and Analysis of Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2308.15126/) (2023)
- [Cross-Modal Attention Calibration for LVLM Hallucination Mitigation](https://mlsanity.com/paper/2501.01926/) (2025)
- [Mitigating Object Hallucinations in Large Vision-Language Models via Attention Calibration](https://mlsanity.com/paper/2502.01969/) (2025)
- [Cross-Image Contrastive Decoding: Precise, Lossless Suppression of Language Priors in Large Vision-Language Models](https://mlsanity.com/paper/2505.10634/) (2025)

## Built on by (most cited first, 8 of 8)

- [Tell Model Where to Look: Mitigating Hallucinations in MLLMs by Vision-Guided Attention](https://mlsanity.com/paper/2511.20032/) (2025)
- [Dismantling Pathological Shortcuts: A Causal Framework for Faithful LVLM Decoding](https://mlsanity.com/paper/2606.27596/) (2026)
- [Towards Mitigating Hallucinations in Large Vision-Language Models by Refining Textual Embeddings](https://mlsanity.com/paper/2511.05017/) (2025)
- [Adversarial Orthogonal Disentanglement for LVLM Hallucination Mitigation](https://mlsanity.com/paper/2605.25377/) (2026)
- [Hallucinations Leave a Grounding Signature:Verifier-Guided Decoding for Selective Object Correction](https://mlsanity.com/paper/2607.27823/) (2026)
- [Information-Regularized Attention for Visual-Centric Reasoning](https://mlsanity.com/paper/2607.00434/) (2026)
- [Role-Break in Attention Heads: Understanding and Detecting Hallucinations in VLMs](https://mlsanity.com/paper/2607.29412/) (2026)
- [GeoPID: Decomposing and Steering Visual Information in Vision-Language Models](https://mlsanity.com/paper/2610.08401/) (2026)

## Abstract (arXiv)

Due to the unidirectional masking mechanism, Decoder-Only models propagate information from left to right. LVLMs (Large Vision-Language Models) follow the same architecture, with visual information gradually integrated into semantic representations during forward propagation. Through systematic analysis, we observe that the majority of the visual information is absorbed into the semantic representations. However, the model's attention distribution does not exhibit sufficient emphasis on semantic representations. This misalignment between the attention distribution and the actual information flow undermines the model's visual understanding ability and contributes to hallucinations. To address this issue, we enhance the model's visual understanding by leveraging the core information embedded in semantic representations. Specifically, we identify attention heads that focus on core semantic representations based on their attention distributions. Then, through a two-stage optimization paradigm, we propagate the advantages of these attention heads across the entire model, aligning the attention distribution with the actual information flow. We evaluate our method on three image captioning benchmarks using five different LVLMs, demonstrating its effectiveness in significantly reducing hallucinations. Further experiments reveal a trade-off between reduced hallucinations and richer details. Notably, our method allows for manual adjustment of the model's conservativeness, enabling flexible control to meet diverse real-world requirements.