mlsanityData to 14 Oct 2026

Visual symbolic mechanisms: Emergent symbol processing in vision language models

arXiv 2506.15871 ↗First posted 18 Jun 2025cs.CVcited by 13

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

To accurately process a visual scene, observers must bind features together to represent individual objects. This capacity is necessary, for instance, to distinguish an image containing a red square and a blue circle from an image containing a blue square and a red circle. Recent work has found that language models solve this 'binding problem' via a set of symbol-like, content-independent indices, but it is unclear whether similar mechanisms are employed by Vision Language Models (VLMs). This question is especially relevant, given the persistent failures of VLMs on tasks that require binding. Here, we identify a previously unknown set of emergent symbolic mechanisms that support binding specifically in VLMs, via a content-independent, spatial indexing scheme. Moreover, we find that binding errors, when they occur, can be traced directly to failures in these mechanisms. Taken together, these results shed light on the mechanisms that support symbol-like processing in VLMs, and suggest possible avenues for reducing the number of binding failures exhibited by these models.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 47 references that are in the map. First posted on arXiv 18 Jun 2025.

Builds on

31 of 47 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024What's in the Image? A Deep-Dive into the Vision of Vision Language Models26 Nov
  2. 2024Understanding the Limits of Vision Language Models Through the Lens of the Binding Problem31 Oct
  3. 2024Locality Alignment Improves Vision-Language Models14 Oct
  4. 2024Towards Interpreting Visual Information Processing in Vision-Language Models9 Oct
  5. 2024KiVA: Kid-inspired Visual Analogies for Testing Large Multimodal Models25 Jul
  6. 2024Vision language models are blind: Failing to translate detailed visual features into words9 Jul
  7. 2024What Do VLMs NOTICE? A Mechanistic Interpretability Pipeline for Gaussian-Noise-free Text-Image Corruption and Evaluation24 Jun
  8. 2024Understanding Information Storage and Transfer in Multi-modal Large Language Models6 Jun

Other references · 23

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  3. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  4. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  5. 2020Object-Centric Learning with Slot Attentioncited by 876
  6. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 610
  7. 2023Function Vectors in Large Language Modelscited by 308
  8. 2013Direct and Indirect Effectscited by 280
  9. 2020On the Binding Problem in Artificial Neural Networkscited by 233
  10. 2023How do Language Models Bind Entities in Context?cited by 108
  11. 2022Does CLIP Bind Concepts? Probing Compositionality in Large Image Modelscited by 102
  12. 2023Comparing Humans, GPT-4, and GPT-4V On Abstraction and Reasoning Taskscited by 55
11 more
  1. 2023Break It Down: Evidence for Structural Compositionality in Neural Networkscited by 44
  2. 2024Monitoring Latent World States in Language Models with Propositional Probescited by 43
  3. 2024Good at captioning, bad at counting: Benchmarking GPT-4V on Earth observation datacited by 39
  4. 2025Emergent Symbolic Mechanisms Support Abstract Reasoning in Large Language Modelscited by 31
  5. 2023Rotating Features for Object Discoverycited by 29
  6. 2023The Relational Bottleneck as an Inductive Bias for Efficient Abstractioncited by 27
  7. 2024Beyond the Doors of Perception: Vision Transformers Represent Relations Between Objectscited by 13
  8. 2025Object-centric Binding in Contrastive Language-Image Pretrainingcited by 12
  9. 2025An explainable transformer circuit for compositional generalizationcited by 8
  10. 2025Evaluating Compositional Scene Understanding in Multimodal Generative Modelscited by 5
  11. 2021The Structure of Systematicity in the Braincited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 12.

13 papers.

Sort:

By area:

  1. 2026The Geometry of Representational Failures in Vision Language Modelscited by 6
  2. 2026Gaze Heads: How VLMs Look at What They Describecited by 5
  3. 2025Uncovering Grounding IDs: How External Cues Shape Multimodal Bindingcited by 4
  4. 2026The Dual Mechanisms of Spatial Variable Binding in Vision-Language Modelscited by 4
  5. 2026I Walk the Line: Examining the Role of Gestalt Continuity in Object Binding for Vision Transformerscited by 1
  6. 2026Uncovering and Shaping the Latent Representation of 3D Scene Topology in Vision-Language Modelscited by 1
  7. 2026Who Says What: Symbolic Trimodal Binding Mechanisms in Audio-Visual LLMscited by 1
  8. 2026VLMs Trace Without Tracking: Diagnosing Failures in Visual Path Followingcited by 0
  9. 2026Binding Visual Features Point by Pointcited by 0
  10. 2026How can embedding models bind concepts?cited by 0
  11. 2026Formalizing the Binding Problemcited by 0
  12. 2026From Interpretability Methods to Interpretable Modelscited by 0
1 more
  1. 2026Matching Object or Relation? Tracing Abstract Reasoning Inside VLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Synthesizing Visual Concepts as Vision-Language Programs
  2. 2026Linear Mechanisms for Spatiotemporal Reasoning in Vision Language Models
  3. 2025Investigating Mechanisms for In-Context Vision Language Binding
  4. 2026Symbolic Grounding Reveals Representational Bottlenecks in Abstract Visual Reasoning
  5. 2025Augmented Vision-Language Models: A Systematic Review
  6. 2026Seeing Beyond Redundancy: Task Complexity's Role in Vision Token Specialization in VLLMs
  7. 2025Concept-RuleNet: Grounded Multi-Agent Neurosymbolic Reasoning in Vision Language Models
  8. 2026Pathways of Visual Information Flow in Vision-Language Models