# Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation

arXiv: https://arxiv.org/abs/2602.24041 · HTML: https://mlsanity.com/paper/2602.24041/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 53 of its 62 references that are in the map.
- First posted on arXiv 27 Feb 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Hallucination mitigation in LVLMs](https://mlsanity.com/mllm/hallucination-mitigation-in-lvlms/)

## Builds on (most cited first, 30 of 53)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Tasks](https://mlsanity.com/paper/2312.14238/) (2023)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2306.13394/) (2023)
- [A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge](https://mlsanity.com/paper/2206.01718/) (2022)
- [Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding](https://mlsanity.com/paper/2311.16922/) (2023)
- [Object Hallucination in Image Captioning](https://mlsanity.com/paper/1809.02156/) (2018)
- [Analyzing and Mitigating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2310.00754/) (2023)
- [Detecting and Preventing Hallucinations in Large Vision Language Models](https://mlsanity.com/paper/2308.06394/) (2023)
- [Woodpecker: Hallucination Correction for Multimodal Large Language Models](https://mlsanity.com/paper/2310.16045/) (2023)
- [Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens](https://mlsanity.com/paper/2411.16724/) (2024)
- [On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectification](https://mlsanity.com/paper/2508.05629/) (2025)
- [Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Models](https://mlsanity.com/paper/2410.03577/) (2024)
- [KRIS-Bench: Benchmarking Next-Level Intelligent Image Editing Models](https://mlsanity.com/paper/2505.16707/) (2025)
- [Unlearning Concepts in Diffusion Model via Concept Domain Correction and Concept Preserving Gradient](https://mlsanity.com/paper/2405.15304/) (2024)
- [Number it: Temporal Grounding Videos like Flipping Manga](https://mlsanity.com/paper/2411.10332/) (2024)
- [Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Key](https://mlsanity.com/paper/2501.09695/) (2025)
- [Exploring Diverse In-Context Configurations for Image Captioning](https://mlsanity.com/paper/2305.14800/) (2023)
- [Unified Optimal Transport Framework for Universal Domain Adaptation](https://mlsanity.com/paper/2210.17067/) (2022)
- [Logical Closed Loop: Uncovering Object Hallucinations in Large Vision-Language Models](https://mlsanity.com/paper/2402.11622/) (2024)
- [Nullu: Mitigating Object Hallucinations in Large Vision-Language Models via HalluSpace Projection](https://mlsanity.com/paper/2412.13817/) (2024)
- [Iterative label cleaning for transductive and semi-supervised few-shot learning](https://mlsanity.com/paper/2012.07962/) (2020)
- [ClearSight: Visual Signal Enhancement for Object Hallucination Mitigation in Multimodal Large language Models](https://mlsanity.com/paper/2503.13107/) (2025)
- [Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimization](https://mlsanity.com/paper/2405.15356/) (2024)

## Built on by (most cited first, 7 of 7)

- [Mitigating Hallucinations in Large Vision-Language Models without Performance Degradation](https://mlsanity.com/paper/2604.20366/) (2026)
- [Principled Steering via Null-space Projection for Jailbreak Defense in Vision-Language Models](https://mlsanity.com/paper/2603.22094/) (2026)
- [When Looking Is Not Enough: Visual Attention Structure Reveals Hallucination in MLLMs](https://mlsanity.com/paper/2605.11559/) (2026)
- [Through the Lens of Character: Resolving Modality-Role Interference in Multimodal Role-Playing Agent](https://mlsanity.com/paper/2605.09443/) (2026)
- [Rethinking Visual Neglect: Steering via Context-Preference for MLLM Hallucination Mitigation](https://mlsanity.com/paper/2605.27993/) (2026)
- [PEA-DPO: Perception-Enhanced Alignment Direct Preference Optimization for MLLMs Alignment](https://mlsanity.com/paper/2608.19598/) (2026)
- [Dynamic Alignment Compensation for Hallucination Mitigation in Large Vision-Language Models](https://mlsanity.com/paper/2608.28058/) (2026)

## Abstract (arXiv)

Multimodal large language models (MLLMs) have achieved remarkable progress in vision-language reasoning, yet they remain vulnerable to hallucination, where generated content deviates from visual evidence. Existing mitigation strategies either require costly supervision during training or introduce additional latency at inference time. Recent vision enhancement methods attempt to address this issue by reinforcing visual tokens during decoding, but they typically inject all tokens indiscriminately, which causes interference from background regions and distracts the model from critical cues. To overcome this challenge, we propose Adaptive Visual Reinforcement (AIR), a training-free framework for MLLMs. AIR consists of two components. Prototype-based token reduction condenses the large pool of visual tokens into a compact subset to suppress redundancy. OT-guided patch reinforcement quantifies the alignment between hidden states and patch embeddings to selectively integrate the most consistent patches into feed-forward layers. As a result, AIR enhances the model's reliance on salient visual information and effectively mitigates hallucination. Extensive experiments across representative MLLMs demonstrate that AIR substantially reduces hallucination while preserving general capabilities, establishing it as an effective solution for building reliable MLLMs.