# Attention to details, logits to truth: visual-aware attention and logits enhancement to mitigate hallucinations in LVLMs

arXiv: https://arxiv.org/abs/2602.09521 · HTML: https://mlsanity.com/paper/2602.09521/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 20 of its 22 references that are in the map.
- First posted on arXiv 10 Feb 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Hallucination mitigation in LVLMs](https://mlsanity.com/mllm/hallucination-mitigation-in-lvlms/)

## Builds on (most cited first, 20 of 20)

- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [Accelerating Large Language Model Decoding with Speculative Sampling](https://mlsanity.com/paper/2302.01318/) (2023)
- [Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding](https://mlsanity.com/paper/2311.16922/) (2023)
- [Object Hallucination in Image Captioning](https://mlsanity.com/paper/1809.02156/) (2018)
- [OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation](https://mlsanity.com/paper/2311.17911/) (2023)
- [Mitigating Hallucination in Large Multi-Modal Models via Robust Instruction Tuning](https://mlsanity.com/paper/2306.14565/) (2023)
- [Multi-Modal Hallucination Control by Visual Information Grounding](https://mlsanity.com/paper/2403.14003/) (2024)
- [Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding](https://mlsanity.com/paper/2403.18715/) (2024)
- [Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs](https://mlsanity.com/paper/2407.21771/) (2024)
- [Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens](https://mlsanity.com/paper/2411.16724/) (2024)
- [Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Models](https://mlsanity.com/paper/2410.03577/) (2024)
- [ClearSight: Visual Signal Enhancement for Object Hallucination Mitigation in Multimodal Large language Models](https://mlsanity.com/paper/2503.13107/) (2025)
- [Alleviating Hallucination in Large Vision-Language Models with Active Retrieval Augmentation](https://mlsanity.com/paper/2408.00555/) (2024)
- [Castling-ViT: Compressing Self-Attention via Switching Towards Linear-Angular Attention at Vision Transformer Inference](https://mlsanity.com/paper/2211.10526/) (2022)
- [Cross-Modal Attention Calibration for LVLM Hallucination Mitigation](https://mlsanity.com/paper/2501.01926/) (2025)
- [Attention Reallocation: Towards Zero-cost and Controllable Hallucination Mitigation of MLLMs](https://mlsanity.com/paper/2503.08342/) (2025)
- [Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in Multimodal Large Language Models](https://mlsanity.com/paper/2502.01419/) (2025)
- [FilterRAG: Zero-Shot Informed Retrieval-Augmented Generation to Mitigate Hallucinations in VQA](https://mlsanity.com/paper/2502.18536/) (2025)

## Abstract (arXiv)

Existing Large Vision-Language Models (LVLMs) exhibit insufficient visual attention, leading to hallucinations. To alleviate this problem, some previous studies adjust and amplify visual attention. These methods present a limitation that boosting attention for all visual tokens inevitably increases attention to task irrelevant tokens. To tackle this challenge, we propose a training free attentional intervention algorithm to enhance the attention of task-relevant tokens based on the argument that task-relevant tokens generally demonstrate high visual-textual similarities. Specifically, the vision-text cross-attention submatrices, which represent visual-textual correlations, are extracted to construct the reweighting matrices to reallocate attention. Besides, to enhance the contribution of visual tokens, we inject visual attention values into the beam search decoding to identify solutions with higher visual attention. Extensive experiments demonstrate that this method significantly reduces hallucinations across mainstream LVLMs, while preserving the accuracy and coherence of generated content.