mlsanityData to 7 Oct 2026

SAVAA: Mitigating Hallucinations in LVLMs via Step-wise Adaptive Visual Attention Amplification

arXiv 2602.13600 ↗First posted 14 Feb 2026cs.CVcited by 2

On the map: Multimodal LLMs / vision-language models → Hallucination mitigation in LVLMs

Abstract (arXiv)

A line of recent training-free methods for mitigating hallucinations in large vision-language models (LVLMs) operates by amplifying attention to visual tokens during autoregressive generation within a single forward pass. We refer to this paradigm as visual attention amplification (VAA). In this paper, we identify a dual failure pattern in existing VAA methods caused by their use of a fixed amplification factor across generation steps: it can be too weak at some steps, leaving hallucinations unresolved, while too strong at others, introducing new hallucinations. Motivated by this finding, we propose Step-wise Adaptive Visual Attention Amplification (SAVAA), a new VAA framework that estimates hallucination risk for each generated token and uses the estimated risk to adaptively amplify visual attention at the next generation step. Specifically, we introduce Visual Grounding Entropy (VGE), a lightweight hallucination-risk estimator that augments predictive entropy with visual grounding, assigning higher risk to tokens that are uncertain, weakly grounded in the image, or both. Guided by VGE, SAVAA uses the estimated risk to calibrate the VAA factor for the next generation step, applying stronger amplification to higher-risk steps and weaker amplification to lower-risk steps. Across LLaVA-NeXT-7B, Qwen3-VL-8B, and InternVL3.5-8B, SAVAA significantly outperforms baseline methods on generative hallucination benchmarks such as CHAIR, SHR and AMBER. Code is available at: https://github.com/JiachengZ01/SAVVA.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Hallucination mitigation in LVLMs, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 41 of its 50 references that are in the map. First posted on arXiv 14 Feb 2026.

Builds on

41 of 50 references.

Closest ancestors · 23

Same direction, newest first

  1. 2025Mitigating Object Hallucinations via Sentence-Level Early Intervention16 Jul
  2. 2025Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs26 May
  3. 2025Image Tokens Matter: Mitigating Hallucination in Discrete Tokenizer-based Large Vision-Language Models via Latent Editing24 May
  4. 2025More Thinking, Less Seeing? Assessing Amplified Hallucination in Multimodal Reasoning Models23 May
  5. 2025Generate, but Verify: Reducing Hallucination in Vision-Language Models with Retrospective Resampling17 Apr
  6. 2025Decoupling Contrastive Decoding: Robust Hallucination Mitigation in Multimodal Large Language Models9 Apr
  7. 2025TARAC: Mitigating Hallucination in LVLMs via Temporal Attention Real-time Accumulative Connection5 Apr
  8. 2025ClearSight: Visual Signal Enhancement for Object Hallucination Mitigation in Multimodal Large language Models17 Mar
  9. 2025Hallucinatory Image Tokens: A Training-free EAZY Approach on Detecting and Mitigating Object Hallucinations in LVLMs10 Mar
  10. 2025Exploring Causes and Mitigation of Hallucinations in Large Vision Language Models24 Feb
  11. 2025LanP: Rethinking the Impact of Language Priors in Large Vision-Language Models17 Feb
  12. 2024Mitigating Object Hallucination via Concentric Causal Attention21 Oct
11 more
  1. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Models4 Aug
  2. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMs31 Jul
  3. 2024Hallucination of Multimodal Large Language Models: A Survey29 Apr
  4. 2024Mitigating Hallucinations in Large Vision-Language Models with Instruction Contrastive Decoding27 Mar
  5. 2024A Survey on Hallucination in Large Vision-Language Models1 Feb
  6. 2023Hallucination Augmented Contrastive Learning for Multimodal Large Language Model12 Dec
  7. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocation29 Nov
  8. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding28 Nov
  9. 2023Beyond Hallucinations: Enhancing LVLMs through Hallucination-Aware Direct Preference Optimization28 Nov
  10. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation13 Nov
  11. 2023Evaluating Object Hallucination in Large Vision-Language Models17 May

Other references · 18

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  3. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  4. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  5. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  6. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  7. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  8. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  9. 2018Object Hallucination in Image Captioningcited by 661
  10. 2024Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similaritiescited by 156
  11. 2023Enhancing Uncertainty-Based Hallucination Detection with Stronger Focuscited by 98
  12. 2024HaloScope: Harnessing Unlabeled LLM Generations for Hallucination Detectioncited by 96
6 more
  1. 2023Knowing What LLMs DO NOT Know: A Simple Yet Effective Self-Detection Methodcited by 51
  2. 2024Debiasing Multimodal Large Language Models via Penalization of Language Priorscited by 36
  3. 2024Looking Beyond Text: Reducing Language bias in Large Vision-Language Models via Multimodal Dual-Attention and Soft-Image Guidancecited by 32
  4. 2024Seeing the Image: Prioritizing Visual Correlation by Contrastive Alignmentcited by 19
  5. 2025Uncertainty Quantification for Hallucination Detection in Large Language Models: Foundations, Methodology, and Future Directionscited by 10
  6. 2025Tell Model Where to Look: Mitigating Hallucinations in MLLMs by Vision-Guided Attentioncited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Don't Blink: Evidence Collapse during Multimodal Reasoningcited by 2
  2. 2026Hallucinations Leave a Grounding Signature:Verifier-Guided Decoding for Selective Object Correctioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026AdaIAT: Adaptively Increasing Attention to Generated Text to Alleviate Hallucinations in LVLM
  2. 2026Attention to details, logits to truth: visual-aware attention and logits enhancement to mitigate hallucinations in LVLMs
  3. 2025Not All Tokens and Heads Are Equally Important: Dual-Level Attention Intervention for Hallucination Mitigation
  4. 2025VEGAS: Mitigating Hallucinations in Large Vision-Language Models via Vision-Encoder Attention Guided Adaptive Steering
  5. 2024Reducing Hallucinations in Vision-Language Models via Latent Space Steering
  6. 2026MHSA: A Lightweight Framework for Mitigating Hallucinations via Steered Attention in LVLMs
  7. 2026Mitigating Object Hallucinations in LVLMs via Attention Imbalance Rectification
  8. 2026Segmentation-Based Attention Entropy: Detecting and Mitigating Object Hallucinations in Large Vision-Language Models