# PAINT: Paying Attention to INformed Tokens to Mitigate Hallucination in Large Vision-Language Model

arXiv: https://arxiv.org/abs/2501.12206 · HTML: https://mlsanity.com/paper/2501.12206/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 14 of its 47 references that are in the map.
- First posted on arXiv 21 Jan 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Hallucination mitigation in LVLMs](https://mlsanity.com/mllm/hallucination-mitigation-in-lvlms/)

## Builds on (most cited first, 14 of 14)

- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Models](https://mlsanity.com/paper/2304.10592/) (2023)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [A Survey on Multimodal Large Language Models](https://mlsanity.com/paper/2306.13549/) (2023)
- [Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic](https://mlsanity.com/paper/2306.15195/) (2023)
- [Object Hallucination in Image Captioning](https://mlsanity.com/paper/1809.02156/) (2018)
- [LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model](https://mlsanity.com/paper/2304.15010/) (2023)
- [LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models](https://mlsanity.com/paper/2306.09265/) (2023)
- [Evaluation and Analysis of Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2308.15126/) (2023)
- [Negative Object Presence Evaluation (NOPE) to Measure Object Hallucination in Vision-Language Models](https://mlsanity.com/paper/2310.05338/) (2023)
- [Visual Instruction Tuning towards General-Purpose Multimodal Model: A Survey](https://mlsanity.com/paper/2312.16602/) (2023)

## Built on by (most cited first, 8 of 8)

- [Journey Before Destination: On the importance of Visual Faithfulness in Slow Thinking](https://mlsanity.com/paper/2512.12218/) (2025)
- [Towards Mitigating Hallucinations in Large Vision-Language Models by Refining Textual Embeddings](https://mlsanity.com/paper/2511.05017/) (2025)
- [ViKey: Enhancing Temporal Understanding in Videos via Visual Prompting](https://mlsanity.com/paper/2603.23186/) (2026)
- [Addressing Exacerbated Attention Sink for Source-Free Cross-Domain Few-Shot Learning](https://mlsanity.com/paper/2605.25799/) (2026)
- [Conscious Gaze: Adaptive Attention Mechanisms for Hallucination Mitigation in Vision-Language Models](https://mlsanity.com/paper/2512.05546/) (2025)
- [Supervised Fine-Tuning Needs to Unlock the Potential of Token Priority](https://mlsanity.com/paper/2602.01227/) (2026)
- [Mitigating Mask Prior Drift and Positional Attention Collapse in Large Diffusion Vision-Language Models](https://mlsanity.com/paper/2605.14530/) (2026)
- [Detecting Hallucination in LLMs: Tracing the Topological Signatures of Impaired Context Sharing](https://mlsanity.com/paper/2609.21096/) (2026)

## Abstract (arXiv)

Large Vision Language Models (LVLMs) have demonstrated remarkable capabilities in understanding and describing visual content, achieving state-of-the-art performance across various vision-language tasks. However, these models often generate descriptions containing objects or details that are absent in the input image, a phenomenon commonly known as hallucination. Our work investigates the key reasons behind this issue by analyzing the pattern of self-attention in transformer layers. We find that hallucinations often arise from the progressive weakening of attention weight to visual tokens in the deeper layers of the LLM. Some previous works naively boost the attention of all visual tokens to mitigate this issue, resulting in suboptimal hallucination reduction. To address this, we identify two critical sets of visual tokens that facilitate the transfer of visual information from the vision encoder to the LLM. Local tokens encode grounded information about objects present in an image, while summary tokens capture the overall aggregated representation of the image. Importantly, these two sets of tokens require different levels of weight enhancement. To this end, we propose PAINT (Paying Attention to INformed Tokens), a plug-and-play framework that intervenes in the self-attention mechanism of the LLM, selectively boosting the attention weights of local and summary tokens with experimentally learned margins. Evaluation on the MSCOCO image captioning dataset demonstrate that our approach reduces hallucination rates by up to 62.3% compared to baseline models while maintaining accuracy. Code is available at hrefhttps://github.com/hasanar1f/PAINThttps://github.com/hasanar1f/PAINT