mlsanityData to 7 Oct 2026

When Can Text Replace Vision? Structural Bottlenecks in Diagram Reasoning

arXiv 2609.39142 ↗First posted 30 Sep 2026cs.CVcited by 0

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Can structured text replace vision for diagram reasoning? A wrong answer after textualization can arise because the representation omits information the question needs, or because the solver fails to use information that is present. We introduce a diagnostic protocol to distinguish these explanations. Using the same solver model and generation settings, we compare three input conditions: the original image, question-blind structure extracted by a vision-language model, or gold structure derived from the diagram source. Validity-triggered recovery tests truncation and schema failure, question-relevant fidelity measures preservation of answer-critical structure, and matched edge interventions test the effect of error location. On a reserved holdout of 240 public FlowGen diagrams, evaluated under a frozen protocol, gold structure reaches 87% accuracy while direct vision and learned text both remain below 30%. The aggregate comparison includes source-derived relation labels that may not be printed in the image and uses different learned and gold graph encodings, so it does not isolate extraction error alone. Retrying only invalid extractions makes nearly every public representation schema-valid yet leaves accuracy essentially unchanged. The public learned-text deficit relative to gold more than doubles with structural difficulty. Question-relevant topology predicts correctness better than whole-graph topology. In an exposed intervention study, a single answer-relevant edge edit reduces the primary solver's original-answer accuracy to near zero, while matched irrelevant edits largely preserve it. Supplied structure requires fewer solving tokens than vision, but learned acquisition removes this advantage at single use. These comparisons motivate evaluating acquired text by the answer-relevant evidence it preserves and by the solver's ability to use that representation.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 30 of its 48 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

30 of 48 references.

Closest ancestors · 9

Same direction, newest first

  1. 2026The Expense of Seeing: Attaining Trustworthy Multimodal Reasoning Within the Monolithic Paradigm22 Apr
  2. 2025VisRes Bench: On Evaluating the Visual Reasoning Capabilities of VLMs24 Dec
  3. 2025SciVerse: Unveiling the Knowledge Comprehension and Visual Reasoning of LMMs on Multi-modal Scientific Problems13 Mar
  4. 2024Probing Visual Language Priors in VLMs31 Dec
  5. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct
  6. 2022DePlot: One-shot visual language reasoning by plot-to-table translation20 Dec
  7. 2022MatCha: Enhancing Visual Language Pretraining with Math Reasoning and Chart Derendering19 Dec
  8. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering20 Sep
  9. 2022Socratic Models: Composing Zero-Shot Multimodal Reasoning with Language1 Apr

Other references · 21

  1. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  2. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  3. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  4. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  5. 2017Don't Just Assume; Look and Answer: Overcoming Priors for Visual Question Answeringcited by 337
  6. 2019PlotQA: Reasoning over Scientific Plotscited by 317
  7. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  8. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 246
  9. 2024ChemVLM: Exploring the Power of Multimodal Large Language Models in Chemistry Areacited by 68
  10. 2025Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMscited by 48
  11. 2019ICDAR 2019 Competition on Scene Text Visual Question Answeringcited by 46
  12. 2024EXAMS-V: A Multi-Discipline Multilingual Multimodal Exam Benchmark for Evaluating Vision Language Modelscited by 41
9 more
  1. 2023Reading and Reasoning over Chart Images for Evidence-based Automated Fact-Checkingcited by 34
  2. 2026Stop Comparing LLM Agents Without Disclosing the Harnesscited by 25
  3. 2021Classification-Regression for Chart Comprehensioncited by 11
  4. 2025Evaluating Large Language Models on Multimodal Chemistry Olympiad Examscited by 5
  5. 2026Visual Exclusivity Attacks: Automatic Multimodal Red Teaming via Agentic Planningcited by 4
  6. 2026Dynamic Hub-and-Spoke Memory for Streaming Video Understandingcited by 4
  7. 2026DISSECT: Diagnosing Where Vision Ends and Language Priors Begin in Scientific VLMscited by 2
  8. 2026OmniMapBench: Benchmarking Visual-Centric Reasoning on Diverse Map Documentscited by 2
  9. 2026RiVaT-Fuse: Reliability-Calibrated Variational Tensor Fusion with Matrix-Valued Trust for Multimodal Prediction under Modality Uncertaintycited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026DRAGON: A Benchmark for Evidence-Grounded Visual Reasoning over Diagrams
  2. 2025Structure-aware Contrastive Learning for Diagram Understanding of Multimodal Models
  3. 2026Pseudo Contrastive Learning for Diagram Comprehension in Multimodal Models
  4. 2025Math Blind: Failures in Diagram Understanding Undermine Reasoning in MLLMs
  5. 2025ChartMuseum: Testing Visual Reasoning Capabilities of Large Vision-Language Models
  6. 2025Chimera: Diagnosing Shortcut Learning in Visual-Language Understanding
  7. 2024Do Vision-Language Models Really Understand Visual Language?
  8. 2026Evaluating Reasoning Fidelity in Visual Text Generation