mlsanityData to 14 Oct 2026

Gaze Heads: How VLMs Look at What They Describe

arXiv 2606.14703 ↗First posted 12 Jun 2026cs.CV cs.CL cs.LGcited by 5

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

How a vision-language model internally solves the task of describing an image is far from obvious. We find that the model develops a specific mechanism for this: a small set of attention heads in its language-model backbone, which we call gaze heads, whose attention tracks the image region the model is currently describing. We find them with a simple correlation score from a few forward passes, using comic strips as a controlled testbed where narrative order is laid out spatially. These gaze heads do not just track the image tokens being described: redirecting their attention to a chosen region forces the VLM to describe that region instead. A single attention-mask intervention on the top-100 gaze heads, fewer than 9% of all heads, steers the model's answer to any chosen comic panel at 83.1% accuracy, while the same intervention on random heads fails to redirect the answer, and intervening on all heads destroys generation. The same lever also extends to continuous control: switching the gaze target mid-generation makes the model wrap up its current panel description and move to the new one within a few tokens. Beyond comics, the same intervention redirects answers to chosen regions in natural COCO images. The mechanism further recurs across model sizes from 2B to 32B parameters and across other VLM architectures, although some frozen-encoder families show no comparable head set. More broadly, this shows that targeted edits identified through mechanistic analysis can serve as practical inference-time levers for steering multimodal model behavior, without any retraining. Our code, interactive demo, and datasets are available at https://gaze.baulab.info/

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 43 references that are in the map. First posted on arXiv 12 Jun 2026.

Builds on

35 of 43 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025Investigating Spatial Attention Bias in Vision-Language Models20 Dec
  2. 2025Visual symbolic mechanisms: Emergent symbol processing in vision language models18 Jun
  3. 2025Your Large Vision-Language Model Only Needs A Few Attention Heads For Visual Grounding8 Mar
  4. 2024Performance Gap in Entity Knowledge Extraction Across Modalities in Vision Language Models18 Dec
  5. 2024Towards Interpreting Visual Information Processing in Vision-Language Models9 Oct
  6. 2024One missing piece in Vision and Language: A Survey on Comics Understanding14 Sep
  7. 2024What Do VLMs NOTICE? A Mechanistic Interpretability Pipeline for Gaussian-Noise-free Text-Image Corruption and Evaluation24 Jun
  8. 2024Understanding Information Storage and Transfer in Multi-modal Large Language Models6 Jun
  9. 2023Voila-A: Aligning Vision-Language Models with User's Gaze Attention22 Dec

Other references · 26

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  4. 2025Qwen3-VL Technical Reportcited by 2,793
  5. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,510
  6. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  7. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  8. 2022In-context Learning and Induction Headscited by 964
  9. 2023Steering Language Models With Activation Engineeringcited by 922
  10. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  11. 2019Are Sixteen Heads Really Better than One?cited by 816
  12. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 724
14 more
  1. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 706
  2. 2023Interpreting CLIP's Image Representation via Text-Based Decompositioncited by 196
  3. 2024Ovis: Structural Embedding Alignment for Multimodal Large Language Modelcited by 169
  4. 2024Efficient Multimodal Learning from Data-centric Perspectivecited by 122
  5. 2025Why Is Spatial Reasoning Hard for VLMs? An Attention Mechanism Perspective on Focus Areascited by 95
  6. 2024Premise Order Matters in Reasoning with Large Language Modelscited by 72
  7. 2016The Amazing Mysteries of the Gutter: Drawing Inferences Between Panels in Comic Book Narrativescited by 42
  8. 2025Perspective-Aware Reasoning in Vision-Language Models via Mental Imagery Simulationcited by 42
  9. 2020Building a Manga Dataset "Manga109" with Annotations for Multimedia Applicationscited by 38
  10. 2024The Manga Whisperer: Automatically Generating Transcriptions for Comicscited by 21
  11. 2024CoMix: A Comprehensive Benchmark for Multi-Task Comic Understandingcited by 16
  12. 2025Gaze-VLM:Bridging Gaze and VLMs through Attention Regularization for Egocentric Understandingcited by 11
  13. 2025MaskCD: Mitigating LVLM Hallucinations by Image Head Masked Contrastive Decodingcited by 6
  14. 2025From Panels to Prose: Generating Literary Narratives from Comicscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Verbalizable Representations Form a Global Workspace in Language Modelscited by 75
  2. 2026Inherited Heads: Audio language models track speakers with their text backbone's attention, and an attention-mass ranking retrieves a different setcited by 0
  3. 2026Using OCR Heads to Verbalize Image Semanticscited by 0
  4. 2026Look Where You Say You're Looking: Self-Grounded Attention for Visual Reasoningcited by 0
  5. 2026What Do Visual-Head Discovery Scores Actually Find? Attention Source, Necessity, and Random-Control Pitfalls in Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024What's in the Image? A Deep-Dive into the Vision of Vision Language Models
  2. 2026Emergent Goal-Directed Attention in Large Vision-Language Models
  3. 2026Learning to See What You Need: Gaze Attention for Multimodal Large Language Models
  4. 2024Unveiling Visual Perception in Language Models: An Attention Head Analysis Approach
  5. 2025VL4Gaze: Unleashing Vision-Language Models for Gaze Following
  6. 2026Visuals Lie, Consistency Speaks: Disentangling Spatial Attention from Reliability in Vision-Language Models
  7. 2025Investigating The Functional Roles of Attention Heads in Vision Language Models: Evidence for Reasoning Modules
  8. 2026Retrieval Heads Meet Vision: Uncovering How VLMs Locate and Extract Visual Information