mlsanityData to 7 Oct 2026

Seeing and Solving Are Not Enough for Vision-Language Models

arXiv 2609.33694 ↗First posted 27 Sep 2026cs.CVcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-language models (VLMs) answer visual questions by combining visual information extraction with downstream problem solving. We investigate a fundamental question: Does an incorrect answer necessarily reflect a failure in visual extraction or problem solving? A model may succeed at both abilities when tested separately yet still fail on the original multimodal question, a distinction that overall answer accuracy cannot reveal. To study this, we perform a question-level empirical analysis across multiple VLMs and visual domains. We define an exactly scorable task state (i.e., the visual information sufficient to solve a question) and use it to test whether the same model can extract the required state, solve the question from the ground-truth state, and answer the original multimodal question. We find that composition failures, where extraction and solving both succeed but direct answering fails, account for 17.7% to 75.6% of direct-answering errors across multiple VLMs and datasets. To address this failure mode, we introduce a simple yet effective method, termed State Realization Tuning (SRT). SRT fine-tunes LoRA adapters attached to the language-model layers while keeping the pretrained VLM weights frozen. It trains the model to output the ground-truth task state before the final answer in a single autoregressive response. SRT improves over standard supervised fine-tuning by 1.7 to 14.1 percentage points and repairs 92.5% to 98.1% of diagnosed composition failures. A single LoRA adapter trained with SRT also improves performance across substantially different task-state structures. Our work shows that having both visual extraction and problem-solving capabilities does not guarantee correct multimodal answering. Requiring the model to first output the visual information needed to solve the question can help bridge this gap.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 25 of its 30 references that are in the map. First posted on arXiv 27 Sep 2026.

Builds on

25 of 30 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models19 May
  2. 2026DISSECT: Diagnosing Where Vision Ends and Language Priors Begin in Scientific VLMs6 Apr
  3. 2025Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMs20 Oct
  4. 2025MathFlow: Enhancing the Perceptual Flow of MLLMs for Visual Mathematical Problems19 Mar
  5. 2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Step15 Nov
  6. 2024Prism: A Framework for Decoupling and Assessing the Capabilities of VLMs20 Jun

Other references · 19

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2025Qwen2.5-VL Technical Reportcited by 5,418
  3. 2015VQA: Visual Question Answeringcited by 2,489
  4. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
  5. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  6. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  7. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,054
  8. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 871
  9. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  10. 2018Neural-Symbolic VQA: Disentangling Reasoning from Vision and Language Understandingcited by 304
  11. 2025MiniCPM-V 4.5: Cooking Efficient MLLMs via Architecture, Data, and Training Recipecited by 174
  12. 2022DePlot: One-shot visual language reasoning by plot-to-table translationcited by 157
7 more
  1. 2026Molmo2: Open Weights and Data for Vision-Language Models with Video Understanding and Groundingcited by 132
  2. 2024Unveiling the Ignorance of MLLMs: Seeing Clearly, Answering Incorrectlycited by 29
  3. 2024MMCOMPOSITION: Revisiting the Compositionality of Pre-trained Vision-Language Modelscited by 29
  4. 2025Unveiling the Compositional Ability Gap in Vision-Language Reasoning Modelcited by 13
  5. 2025Do VLMs Have Bad Eyes? Diagnosing Compositional Failures via Mechanistic Interpretabilitycited by 4
  6. 2025Compose and Fuse: Revisiting the Foundational Bottlenecks in Multimodal Reasoningcited by 3
  7. 2025CrossCheck-Bench: Diagnosing Compositional Failures in Multimodal Conflict Resolutioncited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Seeing is not Enough: Vision-Language Models Perceive Evidence but Fail to Act
  2. 2026Failing to See or Failing to Know? Attributing Errors in Vision-Language Models
  3. 2026From Perception to Integration: Revisiting the Internal Dynamics of Reasoning in Vision-Language Models
  4. 2026How Do VLMs Fail? Vision-Operation Misalignment in Compositional VQA
  5. 2026Discovering Failure Modes in Vision-Language Models using RL
  6. 2026Evaluating VQA in Vision Language Models using Cooperative Principles
  7. 2026From Knowing to Abstaining: Bridging the Representation-Action Gap in Vision-Language Models
  8. 2024Unveiling the Tapestry of Consistency in Large Vision-Language Models