mlsanityData to 7 Oct 2026

Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding

arXiv 2503.06287 ↗First posted 8 Mar 2025cs.CV cs.AIcited by 78

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Visual grounding seeks to localize the image region corresponding to a free-form text description. Recently, the strong multimodal capabilities of Large Vision-Language Models (LVLMs) have driven substantial improvements in visual grounding, though they inevitably require fine-tuning and additional model components to explicitly generate bounding boxes or segmentation masks. However, we discover that a few attention heads in frozen LVLMs demonstrate strong visual grounding capabilities. We refer to these heads, which consistently capture object locations related to text semantics, as localization heads. Using localization heads, we introduce a straightforward and effective training-free visual grounding framework that utilizes text-to-image attention maps from localization heads to identify the target objects. Surprisingly, only three out of thousands of attention heads are sufficient to achieve competitive localization performance compared to existing LVLM-based visual grounding methods that require fine-tuning. Our findings suggest that LVLMs can innately ground objects based on a deep comprehension of the text-image relationship, as they implicitly focus on relevant image regions to generate informative text outputs. All the source codes will be made available to the public.

Key facts. Cited by 78 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 70 of its 82 references that are in the map. First posted on arXiv 8 Mar 2025.

Builds on

70 of 82 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024F-LMM: Grounding Frozen Large Multimodal Models9 Jun
  2. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Models27 Mar
  3. 2024PSALM: Pixelwise SegmentAtion with Large Multi-Modal Model21 Mar
  4. 2023GroundVLP: Harnessing Zero-shot Visual Grounding from Vision-Language Pre-training and Open-Vocabulary Object Detection22 Dec
  5. 2023GSVA: Generalized Segmentation via Multimodal Large Language Models15 Dec
  6. 2023GLaMM: Pixel Grounding Large Multimodal Model6 Nov
  7. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularity11 Oct
  8. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond24 Aug
  9. 2023LISA: Reasoning Segmentation via Large Language Model1 Aug

Other references · 61

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  5. 2023Segment Anythingcited by 8,158
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
  10. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  11. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  12. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
38 more
  1. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  2. 2016Modeling Context in Referring Expressionscited by 1,054
  3. 2023ShareGPT4V: Improving Large Multi-Modal Models with Better Captionscited by 1,000
  4. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  5. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  6. 2023Shikra: Unleashing Multimodal LLM's Referential Dialogue Magiccited by 891
  7. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  8. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 822
  9. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  10. 2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 722
  11. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  12. 2024Yi: Open Foundation Models by 01.AIcited by 706
  13. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  14. 2021CRIS: CLIP-Driven Referring Image Segmentationcited by 353
  15. 2021LAVT: Language-Aware Vision Transformer for Referring Image Segmentationcited by 347
  16. 2023Unleashing Text-to-Image Diffusion Models for Visual Perceptioncited by 266
  17. 2016Segmentation from Natural Language Expressionscited by 257
  18. 2023GRES: Generalized Referring Expression Segmentationcited by 252
  19. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 235
  20. 2022What the DAAM: Interpreting Stable Diffusion Using Cross Attentioncited by 233
  21. 2021Localizing Objects with Self-Supervised Transformers and no Labelscited by 227
  22. 2023SCLIP: Rethinking Self-Attention for Dense Vision-Language Inferencecited by 169
  23. 2022SeqTR: A Simple yet Universal Network for Visual Groundingcited by 138
  24. 2024The Remarkable Robustness of LLMs: Stages of Inference?cited by 138
  25. 2025See What You Are Told: Visual Attention Sink in Large Multimodal Modelscited by 127
  26. 2017Recurrent Multimodal Interaction for Referring Image Segmentationcited by 125
  27. 2021TS-CAM: Token Semantic Coupled Attention Map for Weakly Supervised Object Localizationcited by 122
  28. 2022ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehensioncited by 119
  29. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 117
  30. 2022SegViT: Semantic Segmentation with Plain Vision Transformerscited by 105
  31. 2024Interpreting and Editing Vision-Language Representations to Mitigate Hallucinationscited by 103
  32. 2024Attention Heads of Large Language Models: A Surveycited by 65
  33. 2023Zero-shot Referring Image Segmentation with Global-Local Context Featurescited by 61
  34. 2022Patch-level Representation Learning for Self-supervised Vision Transformerscited by 60
  35. 2021Grounding 'Grounding' in NLPcited by 44
  36. 2023GREC: Generalized Referring Expression Comprehensioncited by 39
  37. 2022Peekaboo: Text to Image Diffusion Models are Zero-Shot Segmentorscited by 34
  38. 2023Ref-Diff: Zero-shot Referring Image Segmentation with Generative Modelscited by 33

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 25; 2026: 53.

78 papers.

Sort:

By area:

  1. 2025Visual Representation Alignment for Multimodal Large Language Modelscited by 40
  2. 2026Image Generators are Generalist Vision Learnerscited by 40
  3. 2025Constructive Distortion: Improving MLLMs with Attention-Guided Image Warpingcited by 13
  4. 2025Towards Cross-View Point Correspondence in Vision-Language Modelscited by 12
  5. 2025How Do Large Vision-Language Models See Text in Image? Unveiling the Distinctive Role of OCR Headscited by 9
  6. 2025Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Temporal Video Groundingcited by 9
  7. 2025Catching the Details: Self-Distilled RoI Predictors for Fine-Grained MLLM Perceptioncited by 8
  8. 2025LatBot: Distilling Universal Latent Actions for Vision-Language-Action Modelscited by 7
  9. 2025Direct Visual Grounding by Directing Attention of Visual Tokenscited by 6
  10. 2026UniVLR: Unifying Text and Vision in Visual Latent Reasoning for Multimodal LLMscited by 6
  11. 2025ReFineVLA: Reasoning-Aware Teacher-Guided Transfer Fine-Tuningcited by 5
  12. 2025Seeing Beyond Words: Self-Supervised Visual Learning for Multimodal Large Language Modelscited by 5
38 more
  1. 2026Q-Zoom: Query-Aware Adaptive Perception for Efficient Multimodal Large Language Modelscited by 5
  2. 2025Attention Misses Visual Risk: Risk-Adaptive Steering for Multimodal Safety Alignmentcited by 4
  3. 2025Tracing and Mitigating Hallucinations in Multimodal LLMs via Dynamic Attention Localizationcited by 3
  4. 2025Reallocating Attention Across Layers to Reduce Multimodal Hallucinationcited by 3
  5. 2025RxnCaption: Reformulating Reaction Diagram Parsing as Visual Prompt Guided Captioningcited by 3
  6. 2025RGBT-GroundBench: Visual Grounding Beyond RGB in Complex Real-World Scenarioscited by 3
  7. 2026Taming SAM3 in the Wild: A Concept Bank for Open-Vocabulary Segmentationcited by 3
  8. 2026Seeing Clearly without Training: Mitigating Hallucinations in Multimodal LLMs for Remote Sensingcited by 3
  9. 2026Your Vision-Language-Action Model Already Has Attention Heads For Path Deviation Detectioncited by 3
  10. 2026PEARL: Geometry Aligns Semantics for Training-Free Open-Vocabulary Semantic Segmentationcited by 3
  11. 2026Beyond the Global Scores: Fine-Grained Token Grounding as a Robust Detector of LVLM Hallucinationscited by 3
  12. 2026Preference Redirection via Attention Concentration: An Attack on Computer Use Agentscited by 3
  13. 2026Entropy-Gradient Grounding: Training-Free Evidence Retrieval in Vision-Language Modelscited by 3
  14. 2026Mechanistic Insights into Functional Sparsity in Multimodal LLMs via CoRe Headscited by 3
  15. 2026Your Model Already Knows: Attention-Guided Safety Filter for Vision-Language-Action Modelscited by 3
  16. 2025When to Think and When to Look: Uncertainty-Guided Lookbackcited by 2
  17. 2025Investigating The Functional Roles of Attention Heads in Vision Language Models: Evidence for Reasoning Modulescited by 2
  18. 2026Head-Aware Visual Cropping: Enhancing Fine-Grained VQA with Attention-Guided Subimagecited by 2
  19. 2026SwiftVLM: Efficient Vision-Language Model Inference via Cross-Layer Token Bypasscited by 2
  20. 2026MI-Pruner: Crossmodal Mutual Information-guided Token Pruner for Efficient MLLMscited by 2
  21. 2026Aligning What Vision-Language Models See and Perceive with Adaptive Information Flowcited by 2
  22. 2026Gaze Heads: How VLMs Look at What They Describecited by 2
  23. 2025OSPO: Object-Centric Self-Improving Preference Optimization for Text-to-Image Generationcited by 1
  24. 2025Decomposed Attention Fusion in MLLMs for Training-Free Video Reasoning Segmentationcited by 1
  25. 2025Improving Visual Discriminability of CLIP for Training-Free Open-Vocabulary Semantic Segmentationcited by 1
  26. 2025MMRPT: MultiModal Reinforcement Pre-Training via Masked Vision-Dependent Reasoningcited by 1
  27. 2026Multimodal In-context Learning for ASR of Low-resource Languagescited by 1
  28. 2026Beyond Static Cropping: Layer-Adaptive Visual Localization and Decoding Enhancementcited by 1
  29. 2026PointAlign: Feature-Level Alignment Regularization for 3D Vision-Language Modelscited by 1
  30. 2026Interpretable Motion-Attentive Maps: Spatio-Temporally Localizing Concepts in Video Diffusion Transformerscited by 1
  31. 2026RealVLG-R1: A Large-Scale Real-World Visual-Language Grounding Benchmark for Robotic Perception and Manipulationcited by 1
  32. 2026Beyond Language: Grounding Referring Expressions with Hand Pointing in Egocentric Visioncited by 1
  33. 2026ReFineVLA: Multimodal Reasoning-Aware Generalist Robotic Policies via Teacher-Guided Fine-Tuningcited by 1
  34. 2026See What I Mean: Aligning Vision and Language Representations for Video Fine-grained Object Understandingcited by 1
  35. 2026Can Retrieval Heads See Images? Multimodal Retrieval Heads in Long-Context Vision-Language Modelscited by 1
  36. 2026The Truth Stays in the Family: Enhancing Contextual Grounding via Inherited Truthful Heads in Model Lineagescited by 1
  37. 2026Knowing Isn't Always Saying: When Do Spatial Encodings Reach Answers in Vision-Language Models?cited by 1
  38. 2025Two Causes, Not One: Rethinking Omission and Fabrication Hallucinations in MLLMscited by 0

50 most cited of 78 All 78 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025GroundingME: Exposing the Visual Grounding Gap in MLLMs through Multi-Dimensional Evaluation
  2. 2025Towards Understanding Visual Grounding in Visual Language Models
  3. 2024MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMs
  4. 2024Learning Visual Grounding from Generative Vision and Language Model
  5. 2025Connecting the Dots: Training-Free Visual Grounding via Agentic Reasoning
  6. 2026Mechanisms of Object Localization in Vision-Language Models
  7. 2024The ART of Composition: Attention-Regularized Training for Compositional Visual Grounding
  8. 2025Hierarchical Contextual Grounding LVLM: Enhancing Fine-Grained Visual-Language Understanding with Robust Grounding