mlsanityData to 7 Oct 2026

How Do VLMs Fail? Vision-Operation Misalignment in Compositional VQA

arXiv 2607.16094 ↗First posted 17 Jul 2026cs.CVcited by 0

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Compositional visual question answering requires Vision-Language Models (VLMs) to execute multiple reasoning operations like object selection, spatial relation resolution, and attribute verification. Despite strong aggregate performance, the mechanistic basis of VLM failures on this task remains underexplored. To address this gap, we analyze vision-operation misalignment in VLMs by examining how failures relate to specific reasoning operations and the internal computational pathways through which they arise and propagate. We introduce an Operation-centric mechanistic framework that decomposes VLM failures by both the reasoning operation where they originate and the internal computational pathway through which they propagate. Our analysis reveals four dominant failure modes: grounding failure, reasoning failure, attribute extraction failure, and language-prior dominance, each characterized by a distinct relationship between visual grounding strength and answer correctness. Through three complementary causal interventions applied across all transformer layers, we find that object-selection failures are associated primarily with feedforward computation, multi-step relational failures with late-layer direct attention, and attribute-extraction failures with answer-position feedforward computation. Validation on VSR further shows that single-step spatial failures are concentrated at object-position encoding, distinguishing them from multi-step relational composition. These findings reveal distinct computational bottlenecks across operation types and provide a principled basis for targeted diagnosis of VLM failures in multimedia reasoning.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 18 of its 32 references that are in the map. First posted on arXiv 17 Jul 2026.

Builds on

18 of 32 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Probing Visual Language Priors in VLMs31 Dec
  2. 2024Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoning27 Nov
  3. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality7 Apr

Other references · 15

  1. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  2. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  3. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  4. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  5. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  6. 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
  7. 2018Object Hallucination in Image Captioningcited by 661
  8. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  9. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 337
  10. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 134
  11. 2024Why are Visually-Grounded Language Models Bad at Image Classification?cited by 101
  12. 2024Information Flow Routes: Automatically Interpreting Language Models at Scalecited by 92
3 more
  1. 2023Towards Vision-Language Mechanistic Interpretability: A Causal Tracing Tool for BLIPcited by 62
  2. 2024What Do VLMs NOTICE? A Mechanistic Interpretability Pipeline for Gaussian-Noise-free Text-Image Corruption and Evaluationcited by 29
  3. 2025Causal Tracing of Object Representations in Large Vision Language Models: Mechanistic Interpretability and Hallucination Mitigationcited by 22

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Seeing and Solving Are Not Enough for Vision-Language Models
  2. 2025Do VLMs Have Bad Eyes? Diagnosing Compositional Failures via Mechanistic Interpretability
  3. 2026Evaluating VQA in Vision Language Models using Cooperative Principles
  4. 2026Failing to See or Failing to Know? Attributing Errors in Vision-Language Models
  5. 2024ConMe: Rethinking Evaluation of Compositional Reasoning for Modern VLMs
  6. 2026Discovering Failure Modes in Vision-Language Models using RL
  7. 2025Generalizing from SIMPLE to HARD Visual Reasoning: Can We Mitigate Modality Imbalance in VLMs?
  8. 2026Composition, Not Conversation: VLMs Lose the Scene, Not the Thread