# SAVAA: Mitigating Hallucinations in LVLMs via Step-wise Adaptive Visual Attention Amplification

arXiv: https://arxiv.org/abs/2602.13600 · HTML: https://mlsanity.com/paper/2602.13600/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 41 of its 50 references that are in the map.
- First posted on arXiv 14 Feb 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Hallucination mitigation in LVLMs](https://mlsanity.com/mllm/hallucination-mitigation-in-lvlms/)

## Builds on (most cited first, 30 of 41)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://mlsanity.com/paper/2508.18265/) (2025)
- [Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation](https://mlsanity.com/paper/2302.09664/) (2023)
- [Aligning Large Multimodal Models with Factually Augmented RLHF](https://mlsanity.com/paper/2309.14525/) (2023)
- [A-OKVQA: A Benchmark for Visual Question Answering using World Knowledge](https://mlsanity.com/paper/2206.01718/) (2022)
- [Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding](https://mlsanity.com/paper/2311.16922/) (2023)
- [Object Hallucination in Image Captioning](https://mlsanity.com/paper/1809.02156/) (2018)
- [OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation](https://mlsanity.com/paper/2311.17911/) (2023)
- [Hallucination of Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2404.18930/) (2024)
- [A Survey on Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2402.00253/) (2024)
- [AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation](https://mlsanity.com/paper/2311.07397/) (2023)
- [Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization](https://mlsanity.com/paper/2311.16839/) (2023)
- [Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding](https://mlsanity.com/paper/2403.18715/) (2024)
- [Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs](https://mlsanity.com/paper/2407.21771/) (2024)
- [Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similarities](https://mlsanity.com/paper/2405.20003/) (2024)
- [Hallucination Augmented Contrastive Learning for Multimodal Large Language Model](https://mlsanity.com/paper/2312.06968/) (2023)
- [Enhancing Uncertainty-Based Hallucination Detection with Stronger Focus](https://mlsanity.com/paper/2311.13230/) (2023)
- [HaloScope: Harnessing Unlabeled LLM Generations for Hallucination Detection](https://mlsanity.com/paper/2409.17504/) (2024)
- [Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models](https://mlsanity.com/paper/2408.02032/) (2024)
- [More Thinking, Less Seeing? Assessing Amplified Hallucination in Multimodal Reasoning Models](https://mlsanity.com/paper/2505.21523/) (2025)
- [Knowing What LLMs DO NOT Know: A Simple Yet Effective Self-Detection Method](https://mlsanity.com/paper/2310.17918/) (2023)
- [Mitigating Object Hallucination via Concentric Causal Attention](https://mlsanity.com/paper/2410.15926/) (2024)
- [ClearSight: Visual Signal Enhancement for Object Hallucination Mitigation in Multimodal Large language Models](https://mlsanity.com/paper/2503.13107/) (2025)
- [Debiasing Multimodal Large Language Models via Penalization of Language Priors](https://mlsanity.com/paper/2403.05262/) (2024)
- [Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resampling](https://mlsanity.com/paper/2504.13169/) (2025)
- [Looking Beyond Text: Reducing Language bias in Large Vision-Language Models via Multimodal Dual-Attention and Soft-Image Guidance](https://mlsanity.com/paper/2411.14279/) (2024)

## Built on by (most cited first, 2 of 2)

- [Don't Blink: Evidence Collapse during Multimodal Reasoning](https://mlsanity.com/paper/2604.04207/) (2026)
- [Hallucinations Leave a Grounding Signature:Verifier-Guided Decoding for Selective Object Correction](https://mlsanity.com/paper/2607.27823/) (2026)

## Abstract (arXiv)

A line of recent training-free methods for mitigating hallucinations in large vision-language models (LVLMs) operates by amplifying attention to visual tokens during autoregressive generation within a single forward pass. We refer to this paradigm as visual attention amplification (VAA). In this paper, we identify a dual failure pattern in existing VAA methods caused by their use of a fixed amplification factor across generation steps: it can be too weak at some steps, leaving hallucinations unresolved, while too strong at others, introducing new hallucinations. Motivated by this finding, we propose Step-wise Adaptive Visual Attention Amplification (SAVAA), a new VAA framework that estimates hallucination risk for each generated token and uses the estimated risk to adaptively amplify visual attention at the next generation step. Specifically, we introduce Visual Grounding Entropy (VGE), a lightweight hallucination-risk estimator that augments predictive entropy with visual grounding, assigning higher risk to tokens that are uncertain, weakly grounded in the image, or both. Guided by VGE, SAVAA uses the estimated risk to calibrate the VAA factor for the next generation step, applying stronger amplification to higher-risk steps and weaker amplification to lower-risk steps. Across LLaVA-NeXT-7B, Qwen3-VL-8B, and InternVL3.5-8B, SAVAA significantly outperforms baseline methods on generative hallucination benchmarks such as CHAIR, SHR and AMBER. Code is available at: https://github.com/JiachengZ01/SAVVA.