mlsanityData to 14 Oct 2026

Linguistic Context Recodes Visual Representations in Vision-Language Models

arXiv 2608.00035 ↗First posted 21 Jul 2026cs.AI cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Goal-directed visual processing is a hallmark of human visual intelligence, resulting in representations that support downstream tasks such as categorization or search. Though vision-language models (VLMs) are often faced with these same tasks, their ability to recode visual representations when presented with goal-directed language remains poorly characterized. Indeed, prior work largely treats visual representations in VLMs as static repositories of visual information that are manipulated by language representations. In the present work, we provide evidence for two concrete instances of language-induced recoding of visual representations. First, we identify an abstract reference representation that denotes which objects are goal-relevant under a natural language prompt. We extract contrastive steering vectors corresponding to this reference representation and demonstrate that they are causally implicated in model predictions. These reference representations are abstract in that they generalize to different objects, different task contexts, and even from synthetic to naturalistic images. Second, we demonstrate language-induced attribute modulation: later layers selectively amplify goal-relevant attributes in visual representations of objects. We demonstrate this phenomenon across a range of different prompts. Finally, we provide a causal intervention that demonstrates that attribute modulation mediates a VLM's response distribution. Together, our results support a more dynamic account of cross-modality processing in VLMs -- rather than vision tokens serving as static repositories of information, they are modulated to support queries articulated in language.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 19 of its 35 references that are in the map. First posted on arXiv 21 Jul 2026.

Builds on

19 of 35 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026The Geometry of Representational Failures in Vision Language Models2 Feb
  2. 2025How Do Vision-Language Models Process Conflicting Information Across Modalities?2 Jul
  3. 2025Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMs10 Jun
  4. 2025Pixels Versus Priors: Controlling Knowledge Priors in Vision-Language Models through Visual Counterfacts21 May
  5. 2025Words or Vision: Do Vision-Language Models Have Blind Faith in Text?4 Mar
  6. 2024Understanding the Limits of Vision Language Models Through the Lens of the Binding Problem31 Oct

Other references · 13

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2015VQA: Visual Question Answeringcited by 2,506
  3. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
  4. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,085
  5. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  6. 2023How do Language Models Bind Entities in Context?cited by 108
  7. 2024The Semantic Hub Hypothesis: Language Models Share Semantic Representations Across Languages and Modalitiescited by 69
  8. 2023Towards Vision-Language Mechanistic Interpretability: A Causal Tracing Tool for BLIPcited by 62
  9. 2025Does Object Binding Naturally Emerge in Large Pretrained Vision Transformers?cited by 19
  10. 2025Visual Representations inside the Language Modelcited by 15
  11. 2024Beyond the Doors of Perception: Vision Transformers Represent Relations Between Objectscited by 13
  12. 2026Cross-Modal Redundancy and the Geometry of Vision-Language Embeddingscited by 10
1 more
  1. 2026I Walk the Line: Examining the Role of Gestalt Continuity in Object Binding for Vision Transformerscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Pathways of Visual Information Flow in Vision-Language Models
  2. 2024If CLIP Could Talk: Understanding Vision-Language Model Representations Through Their Preferred Concept Descriptions
  3. 2025The in-context inductive biases of vision-language models differ across modalities
  4. 2026Mechanisms of Object Localization in Vision-Language Models
  5. 2026Do Vision-Language-Action Models Understand Instructions? A Mechanistic Interpretability Study on Language Grounding
  6. 2025V-SEAM: Visual Semantic Editing and Attention Modulating for Causal Interpretability of Vision-Language Models
  7. 2024Vision-Language Models Align with Human Neural Representations in Concept Processing
  8. 2026Not All Features Are Created Equal: A Mechanistic Study of Vision-Language-Action Models