mlsanityData to 7 Oct 2026

Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in Multimodal Large Language Models

arXiv 2502.01419 ↗First posted 3 Feb 2025cs.CV cs.AIcited by 21

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Detailed image captioning is essential for tasks like data generation and aiding visually impaired individuals. High-quality captions require a balance between precision and recall, which remains challenging for current multimodal large language models (MLLMs). In this work, we hypothesize that this limitation stems from weakening and increasingly noisy visual attention as responses lengthen. To address this issue, we propose SPARC (Selective Progressive Attention ReCalibration), a training-free method that enhances the contribution of visual tokens during decoding. SPARC is founded on three key observations: (1) increasing the influence of all visual tokens reduces recall; thus, SPARC selectively amplifies visual tokens; (2) as captions lengthen, visual attention becomes noisier, so SPARC identifies critical visual tokens by leveraging attention differences across time steps; (3) as visual attention gradually weakens, SPARC reinforces it to preserve its influence. Our experiments, incorporating both automated and human evaluations, demonstrate that existing methods improve the precision of MLLMs at the cost of recall. In contrast, our proposed method enhances both precision and recall with minimal computational overhead.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 48 of its 57 references that are in the map. First posted on arXiv 3 Feb 2025.

Builds on

48 of 57 references.

References · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Visual Instruction Tuningcited by 7,792
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  7. 2024GPT-4o System Cardcited by 4,467
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  10. 2023Qwen Technical Reportcited by 3,273
  11. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  12. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
36 more
  1. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  2. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  3. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  4. 2023Vision Transformers Need Registerscited by 844
  5. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  6. 2023Instruction Tuning with GPT-4cited by 676
  7. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  8. 2018Object Hallucination in Image Captioningcited by 661
  9. 2023OPERA: Alleviating Hallucination in Multi-Modal Large Language Models via Over-Trust Penalty and Retrospection-Allocationcited by 468
  10. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  11. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 352
  12. 2023Analyzing and Mitigating Object Hallucination in Large Vision-Language Modelscited by 314
  13. 2023Detecting and Preventing Hallucinations in Large Vision Language Modelscited by 254
  14. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  15. 2024Multi-Modal Hallucination Control by Visual Information Groundingcited by 196
  16. 2023Efficient Large Language Models: A Surveycited by 183
  17. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMscited by 174
  18. 2023Compositional Chain-of-Thought Prompting for Large Multimodal Modelscited by 160
  19. 2023ChartLlama: A Multimodal LLM for Chart Understanding and Generationcited by 158
  20. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lenscited by 139
  21. 2023Holistic Analysis of Hallucination in GPT-4V(ision): Bias and Interference Challengescited by 131
  22. 2024DOCCI: Descriptions of Connected and Contrasting Imagescited by 130
  23. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decodingcited by 118
  24. 2023Volcano: Mitigating Multimodal Hallucination through Self-Feedback Guided Revisioncited by 96
  25. 2024Self-Introspective Decoding: Alleviating Hallucinations for Large Vision-Language Modelscited by 90
  26. 2024ImageInWords: Unlocking Hyper-Detailed Image Descriptionscited by 56
  27. 2024Seeing Clearly by Layer Two: Enhancing Attention Heads to Alleviate Hallucination in LVLMscited by 49
  28. 2023Multimodal LLMs for health grounded in individual-specific datacited by 48
  29. 2024Understanding Information Storage and Transfer in Multi-modal Large Language Modelscited by 48
  30. 2024Mitigating Object Hallucination via Concentric Causal Attentioncited by 48
  31. 2024DAMRO: Dive into the Attention Mechanism of LVLM to Reduce Object Hallucinationcited by 47
  32. 2024Don't Miss the Forest for the Trees: Attentional Vision Calibration for Large Vision Language Modelscited by 46
  33. 2024Investigating and Mitigating the Multimodal Hallucination Snowballing in Large Vision-Language Modelscited by 31
  34. 2025Cross-Modal Attention Calibration for LVLM Hallucination Mitigationcited by 21
  35. 2024Toward Robust Hyper-Detailed Image Captioning: A Multiagent Approach and Dual Evaluation Metrics for Factuality and Coveragecited by 20
  36. 2023A Multi-Modal Foundation Model to Assist People with Blindness and Low Vision in Environmental Interactioncited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 4; 2026: 17.

21 papers.

Sort:

By area:

  1. 2025EvoLMM: Self-Evolving Large Multimodal Models with Continuous Rewardscited by 24
  2. 2025RSCC: A Large-Scale Remote Sensing Change Caption Dataset for Disaster Eventscited by 13
  3. 2025Uncovering Grounding IDs: How External Cues Shape Multimodal Bindingcited by 4
  4. 2026KVSmooth: Mitigating Hallucination in Multi-modal Large Language Models through Key-Value Smoothingcited by 3
  5. 2026Thinking Diffusion: Penalize and Guide Visual-Grounded Reasoning in Diffusion Multimodal Language Modelscited by 3
  6. 2025Mitigating Visual Context Degradation in Large Multimodal Models: A Training-Free Decoupled Agentic Frameworkcited by 2
  7. 2026StAR: Segment Anything Reasonercited by 2
  8. 2026Focus, Don't Prune: Identifying Instruction-Relevant Regions for Information-Rich Image Understandingcited by 2
  9. 2026Mitigating Hallucinations in Large Vision-Language Models via Causal Route Gatingcited by 2
  10. 2026Addressing Exacerbated Attention Sink for Source-Free Cross-Domain Few-Shot Learningcited by 2
  11. 2026Attention-space Contrastive Guidance for Efficient Hallucination Mitigation in LVLMscited by 1
  12. 2026TraversalBench: Challenging Paths to Follow for Vision Language Modelscited by 1
9 more
  1. 2026Steer Where It Matters: Token-Level Visual-Sensitivity Steering for LVLMs Hallucination Mitigationcited by 1
  2. 2026The Truth Stays in the Family: Enhancing Contextual Grounding via Inherited Truthful Heads in Model Lineagescited by 1
  3. 2026Attention to details, logits to truth: visual-aware attention and logits enhancement to mitigate hallucinations in LVLMscited by 0
  4. 2026Cross-modal Identity Mapping: Minimizing Information Loss in Modality Conversion via Reinforcement Learningcited by 0
  5. 2026HaloProbe: Bayesian Detection and Mitigation of Object Hallucinations in Vision-Language Modelscited by 0
  6. 2026Combating Visual Neglect and Semantic Drift in Large Multimodal Models for Enhanced Cross-Modal Retrievalcited by 0
  7. 2026Brevity is the Soul of Inference Efficiency: Inducing Concision in VLMs via Data Curationcited by 0
  8. 2026When Structured Sparse Autoencoders Learn Consistent Concepts Across Modalitiescited by 0
  9. 2026SAGE: Sink-Aware Guided Emphasis for Visual Grounding in Vision-Language Decoderscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024ST3: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimming
  2. 2025Perceiving Beyond Language Priors: Enhancing Visual Comprehension and Attention in Multimodal Models
  3. 2025Seeing is Understanding: Unlocking Causal Attention into Modality-Mutual Attention for Multimodal LLMs
  4. 2025See What You Are Told: Visual Attention Sink in Large Multimodal Models
  5. 2026Learning to See What You Need: Gaze Attention for Multimodal Large Language Models
  6. 2025Unleashing the Intrinsic Visual Representation Capability of Multimodal Large Language Models
  7. 2024PAR: Prompt-Aware Token Reduction Method for Efficient Large Multimodal Models
  8. 2024Accelerating Multimodal Large Language Models via Dynamic Visual-Token Exit and the Empirical Findings