DISSECT: Diagnosing Where Vision Ends and Language Priors Begin in Scientific VLMs
On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning
Abstract (arXiv)
When asked to describe a molecular diagram, a Vision-Language Model correctly identifies ``a benzene ring with an -OH group.'' When asked to reason about the same image, it answers incorrectly. The model can see but it cannot think about what it sees. We term this the perception-integration gap: a failure where visual information is successfully extracted but lost during downstream reasoning, invisible to single-configuration benchmarks that conflate perception with integration under one accuracy number. To systematically expose such failures, we introduce DISSECT, a 12,000-question diagnostic benchmark spanning Chemistry (7,000) and Biology (5,000). Every question is evaluated under five input modes -- Vision+Text, Text-Only, Vision-Only, Human Oracle, and a novel Model Oracle in which the VLM first verbalizes the image and then reasons from its own description -- yielding diagnostic gaps that decompose performance into language-prior exploitation, visual extraction, perception fidelity, and integration effectiveness. Evaluating 18~VLMs, we find that: (1) Chemistry exhibits substantially lower language-prior exploitability than Biology, confirming molecular visual content as a harder test of genuine visual reasoning; (2) Open-source models consistently score higher when reasoning from their own verbalized descriptions than from raw images, exposing a systematic integration bottleneck; and (3) Closed-source models show no such gap, indicating that bridging perception and integration is the frontier separating open-source from closed-source multimodal capability. The Model Oracle protocol is both model and benchmark agnostic, applicable post-hoc to any VLM evaluation to diagnose integration failures.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 8 of its 21 references that are in the map. First posted on arXiv 6 Apr 2026.
Builds on
8 of 21 references.
Closest ancestors · 3
Same direction, newest first
- 2025SciVerse: Unveiling the Knowledge Comprehension and Visual Reasoning of LMMs on Multi-modal Scientific Problems13 Mar
- 2024Probing Visual Language Priors in VLMs31 Dec
- 2024EXAMS-V: A Multi-Discipline Multilingual Multimodal Exam Benchmark for Evaluating Vision Language Models15 Mar
Other references · 5
- 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
- 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
- 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 337
- 2024ChemVLM: Exploring the Power of Multimodal Large Language Models in Chemistry Areacited by 68
- 2025Evaluating Large Language Models on Multimodal Chemistry Olympiad Examscited by 5
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026ChemVLR: Prioritizing Reasoning in Perception for Chemical Vision-Language Understanding
- 2026Seeing is not Enough: Vision-Language Models Perceive Evidence but Fail to Act
- 2026Decoding Scientific Experimental Images: The SPUR Benchmark for Perception, Understanding, and Reasoning
- 2025Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMs
- 2026The Abstraction Gap in Vision-Language Causal Reasoning
- 2026MolSight: A Graph-Aware Vision-Language Model for Unified Chemical Image Understanding
- 2026PerceptionBench: Evaluating Atomic Visual Perception in Multimodal Large Language Models
- 2025CSVQA: A Chinese Multimodal Benchmark for Evaluating STEM Reasoning Capabilities of VLMs