mlsanityData to 7 Oct 2026

The Geometry of Representational Failures in Vision Language Models

arXiv 2602.07025 ↗First posted 2 Feb 2026cs.CV cs.AIcited by 5

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-Language Models (VLMs) exhibit puzzling failures in multi-object visual tasks, such as hallucinating non-existent elements or failing to identify the most similar objects among distractions. While these errors mirror human cognitive constraints, such as the 'Binding Problem', the internal mechanisms driving them in artificial systems remain poorly understood. Here, we propose a mechanistic insight by analyzing the representational geometry of open-weight VLMs (Qwen, InternVL, Gemma), comparing methodologies to distill "concept vectors'' - latent directions encoding visual concepts. We validate our concept vectors via steering interventions that reliably manipulate model behavior in both simplified and naturalistic vision tasks (e.g., forcing the model to perceive a red flower as blue). We observe that the geometric overlap between these vectors strongly correlates with specific error patterns, offering a grounded quantitative framework to understand how internal representations shape model behavior and drive visual failures.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 45 references that are in the map. First posted on arXiv 2 Feb 2026.

Builds on

26 of 45 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Visual symbolic mechanisms: Emergent symbol processing in vision language models18 Jun
  2. 2024Understanding the Limits of Vision Language Models Through the Lens of the Binding Problem31 Oct
  3. 2024Vision language models are blind: Failing to translate detailed visual features into words9 Jul

Other references · 23

  1. 2023Visual Instruction Tuningcited by 7,792
  2. 2025Qwen2.5-VL Technical Reportcited by 5,418
  3. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  4. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  5. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,309
  6. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  7. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  8. 2023Steering Llama 2 via Contrastive Activation Additioncited by 1,009
  9. 2023Steering Language Models With Activation Engineeringcited by 824
  10. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  11. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  12. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
11 more
  1. 2024Hallucination is Inevitable: An Innate Limitation of Large Language Modelscited by 329
  2. 2024Not All Language Model Features Are One-Dimensionally Linearcited by 201
  3. 2024The Geometry of Categorical and Hierarchical Concepts in Large Language Modelscited by 151
  4. 2024Improving Steering Vectors by Targeting Sparse Autoencoder Featurescited by 86
  5. 2025The Origins of Representation Manifolds in Large Language Modelscited by 46
  6. 2026When Models Manipulate Manifolds: The Geometry of a Counting Taskcited by 43
  7. 2020Emergence of Separable Manifolds in Deep Language Representationscited by 39
  8. 2024A Vision Check-up for Language Modelscited by 36
  9. 2018Color naming reflects both perceptual structure and communicative needcited by 13
  10. 2025HEDGE: Hallucination Estimation via Dense Geometric Entropy for VQA with Vision-Language Modelscited by 5
  11. 2025Bound by semanticity: universal laws governing the generalization-identification tradeoffcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026EGA: Adapting Frozen Encoders for Vector Search with Bounded Out-of-Distribution Degradationcited by 0
  2. 2026Probing for Representation Manifolds in Superpositioncited by 0
  3. 2026Decodable Is Not Grounded: A Vision-Ablation Arbiter for VLM Spatial Reasoningcited by 0
  4. 2026The Ebb and Flow of Multimodal Focus: Scheduling Visual Relay Windows for Grounded VLM Reasoningcited by 0
  5. 2026Linguistic Context Recodes Visual Representations in Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Hidden in plain sight: VLMs overlook their visual representations
  2. 2026Semantic Richness or Geometric Reasoning? The Fragility of VLM's Visual Invariance
  3. 2026Seeing Beyond Redundancy: Task Complexity's Role in Vision Token Specialization in VLLMs
  4. 2024UniBench: Visual Reasoning Requires Rethinking Vision-Language Beyond Scaling
  5. 2026Not All Error Yields to Scale: Where Scaling Stops in Vision-Language Inference
  6. 2026Discovering Failure Modes in Vision-Language Models using RL
  7. 2026Investigating Relational Reasoning in VLMs
  8. 2026The Last Visible Pixel: Probing Fine-Scale Perception in Vision-Language Models