mlsanityData to 7 Oct 2026

See What You Are Told: Visual Attention Sink in Large Multimodal Models

arXiv 2503.03321 ↗First posted 5 Mar 2025cs.CV cs.AIcited by 127

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

Abstract (arXiv)

Large multimodal models (LMMs) "see" images by leveraging the attention mechanism between text and visual tokens in the transformer decoder. Ideally, these models should focus on key visual information relevant to the text token. However, recent findings indicate that LMMs have an extraordinary tendency to consistently allocate high attention weights to specific visual tokens, even when these tokens are irrelevant to the corresponding text. In this study, we investigate the property behind the appearance of these irrelevant visual tokens and examine their characteristics. Our findings show that this behavior arises due to the massive activation of certain hidden state dimensions, which resembles the attention sink found in language models. Hence, we refer to this phenomenon as the visual attention sink. In particular, our analysis reveals that removing the irrelevant visual sink tokens does not impact model performance, despite receiving high attention weights. Consequently, we recycle the attention to these tokens as surplus resources, redistributing the attention budget to enhance focus on the image. To achieve this, we introduce Visual Attention Redistribution (VAR), a method that redistributes attention in image-centric heads, which we identify as innately focusing on visual information. VAR can be seamlessly applied across different LMMs to improve performance on a wide range of tasks, including general vision-language tasks, visual hallucination tasks, and vision-centric tasks, all without the need for additional training, models, or inference steps. Experimental results demonstrate that VAR enables LMMs to process visual information more effectively by adjusting their internal attention mechanisms, offering a new direction to enhancing the multimodal capabilities of LMMs.

Key facts. Cited by 127 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 59 of its 69 references that are in the map. First posted on arXiv 5 Mar 2025.

Builds on

59 of 69 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024HiRED: Attention-Guided Token Dropping for Efficient Inference of High-Resolution Vision-Language Models20 Aug
  2. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Models11 Mar

Other references · 57

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  10. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  11. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
  12. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
38 more
  1. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  2. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  3. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  4. 2024Qwen2 Technical Reportcited by 2,076
  5. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  6. 2019Towards VQA Models That Can Readcited by 1,696
  7. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  8. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  9. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  10. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehensioncited by 929
  11. 2023A Survey on Multimodal Large Language Modelscited by 898
  12. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  13. 2023Vision Transformers Need Registerscited by 844
  14. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  15. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  16. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  17. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  18. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  19. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 671
  20. 2018Object Hallucination in Image Captioningcited by 661
  21. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  22. 2023HallusionBench: An Advanced Diagnostic Suite for Entangled Language Hallucination and Visual Illusion in Large Vision-Language Modelscited by 572
  23. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  24. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  25. 2024Massive Activations in Large Language Modelscited by 277
  26. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 200
  27. 2024Multi-Modal Hallucination Control by Visual Information Groundingcited by 196
  28. 2024Paying More Attention to Image: A Training-Free Method for Alleviating Hallucination in LVLMscited by 174
  29. 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 147
  30. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 138
  31. 2024IBD: Alleviating Hallucinations in Large Vision-Language Models via Image-Biased Decodingcited by 118
  32. 2024Towards Interpreting Visual Information Processing in Vision-Language Modelscited by 115
  33. 2024Information Flow Routes: Automatically Interpreting Language Models at Scalecited by 92
  34. 2023Tell Your Model Where to Attend: Post-hoc Attention Steering for LLMscited by 82
  35. 2024Mitigating Object Hallucinations in Large Vision-Language Models with Assembly of Global and Local Attentioncited by 74
  36. 2024Unveiling and Harnessing Hidden Attention Sinks: Enhancing Large Language Models without Training through Attention Calibrationcited by 70
  37. 2024What's in the Image? A Deep-Dive into the Vision of Vision Language Modelscited by 66
  38. 2024Attention Heads of Large Language Models: A Surveycited by 65

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 43; 2026: 84.

127 papers.

Sort:

By area:

  1. 2025Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Groundingcited by 78
  2. 2025LiveVLM: Efficient Online Video Understanding via Streaming-Oriented KV Cache and Retrievalcited by 50
  3. 2025Multimodal Chain of Continuous Thought for Latent-Space Reasoning in Vision-Language Modelscited by 29
  4. 2025A Survey of Multimodal Hallucination Evaluation and Detectioncited by 26
  5. 2025To Sink or Not to Sink: Visual Information Pathways in Large Vision-Language Modelscited by 18
  6. 2025HiDe: Rethinking The Zoom-IN method in High Resolution MLLMs via Hierarchical Decouplingcited by 16
  7. 2025SparseVILA: Decoupling Visual Sparsity for Efficient VLM Inferencecited by 15
  8. 2025MVI-Bench: A Comprehensive Benchmark for Evaluating Robustness to Misleading Visual Inputs in LVLMscited by 14
  9. 2025See&Trek: Training-Free Spatial Prompting for Multimodal Large Language Modelcited by 13
  10. 2025When Semantics Mislead Vision: Mitigating Large Multimodal Models Hallucinations in Scene Text Spotting and Understandingcited by 12
  11. 2025GLSim: Detecting Object Hallucinations in LVLMs via Global-Local Similaritycited by 11
  12. 2026Restoring Linguistic Grounding in VLA Models via Train-Free Attention Recalibrationcited by 11
38 more
  1. 2025Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Temporal Video Groundingcited by 9
  2. 2025Catching the Details: Self-Distilled RoI Predictors for Fine-Grained MLLM Perceptioncited by 8
  3. 2026Hallucination Begins Where Saliency Dropscited by 8
  4. 2025Value-State Gated Attention for Mitigating Extreme-Token Phenomena in Transformerscited by 7
  5. 2025SHIELD: Suppressing Hallucinations In LVLM Encoders via Bias and Vulnerability Defensecited by 7
  6. 2026VLA Knows Its Limits: Adaptive Execution Horizons for Robot Policiescited by 7
  7. 2026Fine-Grained Post-Training Quantization for Large Vision Language Models with Quantization-Aware Integrated Gradientscited by 7
  8. 2025Don't Deceive Me: Mitigating Gaslighting through Attention Reallocation in LMMscited by 6
  9. 2025Rethinking Explainability in the Era of Multimodal AIcited by 6
  10. 2025Direct Visual Grounding by Directing Attention of Visual Tokenscited by 6
  11. 2025Can Vision-Language Models Count? A Synthetic Benchmark and Analysis of Attention-Based Interventionscited by 6
  12. 2025Tell Model Where to Look: Mitigating Hallucinations in MLLMs by Vision-Guided Attentioncited by 6
  13. 2025Revealing Perception and Generation Dynamics in LVLMs: Mitigating Hallucinations via Validated Dominance Correctioncited by 6
  14. 2025The Point, the Vision and the Text: Does Point Cloud Boost Spatial Reasoning of Large Language Models? A Bias-Controlled Studycited by 5
  15. 2025Segmentation as A Plug-and-Play Capability for Frozen Multimodal LLMscited by 5
  16. 2026What Do Visual Tokens Really Encode? Uncovering Sparsity and Redundancy in Multimodal Large Language Modelscited by 5
  17. 2026On the Nature of Attention Sink that Shapes Decoding Strategy in Omni-LLMscited by 5
  18. 2026Q-Zoom: Query-Aware Adaptive Perception for Efficient Multimodal Large Language Modelscited by 5
  19. 2026Large Vision-Language Models Get Lost in Attentioncited by 5
  20. 2026OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyondcited by 5
  21. 2025The Power of One: A Single Example is All it Takes for Segmentation in VLMscited by 4
  22. 2025Attention Misses Visual Risk: Risk-Adaptive Steering for Multimodal Safety Alignmentcited by 4
  23. 2025TRivia: Self-supervised Fine-tuning of Vision-Language Models for Table Recognitioncited by 4
  24. 2026FreeText: Training-Free Text Rendering in Diffusion Transformers via Attention Localization and Spectral Glyph Injectioncited by 4
  25. 2026AMPS: Adaptive Modality Preference Steering via Functional Entropycited by 4
  26. 2026Deeper Thought, Weaker Aim: Understanding and Mitigating Perceptual Impairment during Reasoning in Multimodal Large Language Modelscited by 4
  27. 2026Probing Cross-modal Information Hubs in Audio-Visual LLMscited by 4
  28. 2025Seeing It or Not? Interpretable Vision-aware Latent Steering to Mitigate Object Hallucinationscited by 3
  29. 2025Attention Debiasing for Token Pruning in Vision Language Modelscited by 3
  30. 2025Tracing and Mitigating Hallucinations in Multimodal LLMs via Dynamic Attention Localizationcited by 3
  31. 2025Reallocating Attention Across Layers to Reduce Multimodal Hallucinationcited by 3
  32. 2025Capturing Gaze Shifts for Guidance: Cross-Modal Fusion Enhancement for VLM Hallucination Mitigationcited by 3
  33. 2025Attention Guided Alignment in Efficient Vision-Language Modelscited by 3
  34. 2025SAVE: Sparse Autoencoder-Driven Visual Information Enhancement for Mitigating Object Hallucinationcited by 3
  35. 2026PlaM: Training-Free Plateau-Guided Model Merging for Better Visual Grounding in MLLMscited by 3
  36. 2026KVSmooth: Mitigating Hallucination in Multi-modal Large Language Models through Key-Value Smoothingcited by 3
  37. 2026Surgery: Mitigating Harmful Fine-Tuning for Large Language Models via Attention Sinkcited by 3
  38. 2026Focusing Where Vision Matters: Selective Training for Large Vision Language Models via Visual Information Gaincited by 3

50 most cited of 127 All 127 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Disentangling Semantic Attention from Structural Bias in the Attention Manifold
  2. 2026Q Cache: Visual Attention is Valuable in Less than Half of Decode Layers for Multimodal Large Language Model
  3. 2026Learning to See What You Need: Gaze Attention for Multimodal Large Language Models
  4. 2025Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in Multimodal Large Language Models
  5. 2024Unveiling Visual Perception in Language Models: An Attention Head Analysis Approach
  6. 2026Attention Sinks in Diffusion Transformers: A Causal Analysis
  7. 2024Boosting Multimodal Large Language Models with Visual Tokens Withdrawal for Rapid Inference
  8. 2024ST3: Accelerating Multimodal Large Language Model by Spatial-Temporal Visual Token Trimming