Look Carefully: Adaptive Visual Reinforcements in Multimodal Large Language Models for Hallucination Mitigation
On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs
Abstract (arXiv)
Multimodal large language models (MLLMs) have achieved remarkable progress in vision-language reasoning, yet they remain vulnerable to hallucination, where generated content deviates from visual evidence. Existing mitigation strategies either require costly supervision during training or introduce additional latency at inference time. Recent vision enhancement methods attempt to address this issue by reinforcing visual tokens during decoding, but they typically inject all tokens indiscriminately, which causes interference from background regions and distracts the model from critical cues. To overcome this challenge, we propose Adaptive Visual Reinforcement (AIR), a training-free framework for MLLMs. AIR consists of two components. Prototype-based token reduction condenses the large pool of visual tokens into a compact subset to suppress redundancy. OT-guided patch reinforcement quantifies the alignment between hidden states and patch embeddings to selectively integrate the most consistent patches into feed-forward layers. As a result, AIR enhances the model's reliance on salient visual information and effectively mitigates hallucination. Extensive experiments across representative MLLMs demonstrate that AIR substantially reduces hallucination while preserving general capabilities, establishing it as an effective solution for building reliable MLLMs.
Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 53 of its 62 references that are in the map. First posted on arXiv 27 Feb 2026.
Builds on
53 of 62 references.
Closest ancestors · 19
Same direction, newest first
- 2025Modality Bias in LVLMs: Analyzing and Mitigating Object Hallucination via Attention Lens4 Aug
- 2025Mitigating Object Hallucinations via Sentence-Level Early Intervention16 Jul
- 2025ONLY: One-Layer Intervention Sufficiently Mitigates Hallucinations in Large Vision-Language Models1 Jul
- 2025Mitigating Hallucination in Large Vision-Language Models via Adaptive Attention Calibration27 May
- 2025ClearSight: Visual Signal Enhancement for Object Hallucination Mitigation in Multimodal Large language Models17 Mar
- 2025Mitigating Hallucinations in Large Vision-Language Models with Internal Fact-based Contrastive Decoding3 Feb
- 2025CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs28 Jan
- 2025Mitigating Hallucinations in Large Vision-Language Models via DPO: On-Policy Data Hold the Key16 Jan
- 2024Nullu: Mitigating Object Hallucinations in Large Vision-Language Models via HalluSpace Projection18 Dec
- 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lens23 Nov
- 2024Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Models4 Oct
- 2024Reefknot: A Comprehensive Benchmark for Relation Hallucination Evaluation, Analysis and Mitigation in Multimodal Large Language Models18 Aug
7 more
- 2024Alleviating Hallucinations in Large Vision-Language Models through Hallucination-Induced Optimization24 May
- 2024Logical Closed Loop: Uncovering Object Hallucinations in Large Vision-Language Models18 Feb
- 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
- 2023Woodpecker: Hallucination Correction for Multimodal Large Language Models24 Oct
- 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Models1 Oct
- 2023Detecting and Preventing Hallucinations in Large Vision Language Models11 Aug
- 2023Evaluating Object Hallucination in Large Vision-Language Models17 May
Other references · 34
- 2014Microsoft COCO: Common Objects in Contextcited by 17,608
- 2023Visual Instruction Tuningcited by 7,869
- 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
- 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
- 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
- 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,316
- 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
- 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,141
- 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
- 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
- 2018Object Hallucination in Image Captioningcited by 669
- 2025On the Generalization of SFT: A Reinforcement Learning Perspective with Reward Rectificationcited by 127
22 more
- 2025KRIS-Bench: Benchmarking Next-Level Intelligent Image Editing Modelscited by 71
- 2024Unlearning Concepts in Diffusion Model via Concept Domain Correction and Concept Preserving Gradientcited by 69
- 2024Number it: Temporal Grounding Videos like Flipping Mangacited by 65
- 2023Exploring Diverse In-Context Configurations for Image Captioningcited by 63
- 2022Unified Optimal Transport Framework for Universal Domain Adaptationcited by 57
- 2020Iterative label cleaning for transductive and semi-supervised few-shot learningcited by 40
- 2020Multi-source Domain Adaptation via Weighted Joint Distributions Optimal Transportcited by 28
- 2024AWT: Transferring Vision-Language Models via Augmentation, Weighting, and Transportationcited by 28
- 2023Tuning Multi-mode Token-level Prompt Alignment across Modalitiescited by 26
- 2024Enhancing Zero-Shot Vision Models by Label-Free Prompt Distribution Learning and Bias Correctingcited by 26
- 2025Unsupervised Visual Chain-of-Thought Reasoning via Preference Optimizationcited by 22
- 2023PatchCT: Aligning Patch Set and Label Set with Conditional Transport for Multi-Label Image Classificationcited by 21
- 2024Selective Vision-Language Subspace Projection for Few-shot CLIPcited by 18
- 2025Enhancing CLIP Robustness via Cross-Modality Alignmentcited by 15
- 2022Adaptive Distribution Calibration for Few-Shot Learning with Hierarchical Optimal Transportcited by 14
- 2025Dynamic Multimodal Prototype Learning in Vision-Language Modelscited by 13
- 2025Benchmarking and Bridging Emotion Conflicts for Multimodal Emotion Reasoningcited by 12
- 2024Boosting Few-Shot Learning via Attentive Feature Regularizationcited by 10
- 2024Video Repurposing from User Generated Content: A Large-scale Dataset and Benchmarkcited by 9
- 2024CARE Transformer: Mobile-Friendly Linear Visual Transformer via Decoupled Dual Interactioncited by 5
- 2025Hierarchical Semantic Alignment for Image Clusteringcited by 5
- 2021Few-shot Partial Multi-view Learningcited by 3
Built on by
Citing papers per year: 2026: 7.
7 papers.
- 2026Mitigating Hallucinations in Large Vision-Language Models without Performance Degradationcited by 3
- 2026Principled Steering via Null-space Projection for Jailbreak Defense in Vision-Language Modelscited by 2
- 2026When Looking Is Not Enough: Visual Attention Structure Reveals Hallucination in MLLMscited by 1
- 2026Through the Lens of Character: Resolving Modality-Role Interference in Multimodal Role-Playing Agentcited by 0
- 2026Rethinking Visual Neglect: Steering via Context-Preference for MLLM Hallucination Mitigationcited by 0
- 2026PEA-DPO: Perception-Enhanced Alignment Direct Preference Optimization for MLLMs Alignmentcited by 0
- 2026Dynamic Alignment Compensation for Hallucination Mitigation in Large Vision-Language Modelscited by 0
- 2026Dynamic Alignment Compensation for Hallucination Mitigation in Large Vision-Language Models28 Aug 2026
- 2026PEA-DPO: Perception-Enhanced Alignment Direct Preference Optimization for MLLMs Alignment20 Aug 2026
- 2026Rethinking Visual Neglect: Steering via Context-Preference for MLLM Hallucination Mitigation27 May 2026
- 2026When Looking Is Not Enough: Visual Attention Structure Reveals Hallucination in MLLMs12 May 2026
- 2026Through the Lens of Character: Resolving Modality-Role Interference in Multimodal Role-Playing Agent10 May 2026
- 2026Mitigating Hallucinations in Large Vision-Language Models without Performance Degradation22 Apr 2026
- 2026Principled Steering via Null-space Projection for Jailbreak Defense in Vision-Language Models23 Mar 2026
Similar, not cited
Close in wording, no citation either way
- 2024Cracking the Code of Hallucination in LVLMs with Vision-aware Head Divergence
- 2024VaLiD: Mitigating the Hallucination of Large Vision Language Models by Visual Layer Fusion Contrastive Decoding
- 2026Beyond Visual Enhancement: Adaptive Multi-Context Steering to Mitigate LVLM Hallucinations
- 2026ACT Now: Preempting LVLM Hallucinations via Adaptive Context Integration
- 2026Mitigating Hallucination in Vision-Language Models through Barrier-Regulated Adaptive Closed-form Steering
- 2026Hallucination-aware intermediate representation edit in large vision-language models
- 2025Not All Tokens and Heads Are Equally Important: Dual-Level Attention Intervention for Hallucination Mitigation
- 2025Mitigating Image Captioning Hallucinations in Vision-Language Models