mlsanityData to 14 Oct 2026

Do Vision-Language Models Really Understand Visual Language?

arXiv 2410.00193 ↗First posted 30 Sep 2024cs.CL cs.CVcited by 32

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Visual language is a system of communication that conveys information through symbols, shapes, and spatial arrangements. Diagrams are a typical example of a visual language depicting complex concepts and their relationships in the form of an image. The symbolic nature of diagrams presents significant challenges for building models capable of understanding them. Recent studies suggest that Large Vision-Language Models (LVLMs) can even tackle complex reasoning tasks involving diagrams. In this paper, we investigate this phenomenon by developing a comprehensive test suite to evaluate the diagram comprehension capability of LVLMs. Our test suite uses a variety of questions focused on concept entities and their relationships over a set of synthetic as well as real diagrams across domains to evaluate the recognition and reasoning abilities of models. Our evaluation of LVLMs shows that while they can accurately identify and reason about entities, their ability to understand relationships is notably limited. Further testing reveals that the decent performance on diagram understanding largely stems from leveraging their background knowledge as shortcuts to identify and reason about the relational information. Thus, we conclude that LVLMs have a limited capability for genuine diagram understanding, and their impressive performance in diagram reasoning is an illusion emanating from other confounding factors, such as the background knowledge in the models.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 34 of its 54 references that are in the map. First posted on arXiv 30 Sep 2024.

Builds on

34 of 54 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?21 Mar

Other references · 33

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  4. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  5. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  6. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,729
  7. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  8. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  9. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  10. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,317
  11. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  12. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 894
21 more
  1. 2016A Diagram Is Worth A Dozen Imagescited by 804
  2. 2022Visual Programming: Compositional visual reasoning without trainingcited by 516
  3. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 488
  4. 2023AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluationcited by 300
  5. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 267
  6. 2023Talk like a Graph: Encoding Graphs for Large Language Modelscited by 176
  7. 2023Causal Parrots: Large Language Models May Talk Causality But Are Not Causalcited by 150
  8. 2015Yin and Yang: Balancing and Answering Binary Visual Questionscited by 109
  9. 2023Evaluation and Analysis of Hallucination in Large Vision-Language Modelscited by 99
  10. 2021Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Modelscited by 75
  11. 2023Large Language Models Can be Lazy Learners: Analyze Shortcuts in In-Context Learningcited by 69
  12. 2023Large Language Models are Visual Reasoning Coordinatorscited by 58
  13. 2017ShapeWorld - A new test methodology for multimodal language understandingcited by 37
  14. 2024Are Large Vision Language Models up to the Challenge of Chart Comprehension and Reasoning? An Extensive Investigation into the Capabilities and Limitations of LVLMscited by 30
  15. 2022Doubly Right Object Recognition: A Why Prompt for Visual Rationalescited by 29
  16. 2024FlowVQA: Mapping Multimodal Logic in Visual Question Answering with Flowchartscited by 28
  17. 2024FlowLearn: Evaluating Large Vision-Language Models on Flowchart Understandingcited by 27
  18. 2024Case-Based or Rule-Based: How Do Transformers Do the Math?cited by 26
  19. 2023A Comprehensive Evaluation of GPT-4V on Knowledge-Intensive Visual Question Answeringcited by 23
  20. 2016Semantic Parsing to Probabilistic Programs for Situated Question Answeringcited by 5
  21. 2023Parrot Mind: Towards Explaining the Complex Task Reasoning of Pretrained Large Language Models with Template-Content Structurecited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 13; 2026: 18.

32 papers.

Sort:

By area:

  1. 2024VisOnlyQA: Large Vision Language Models Still Struggle with Visual Perception of Geometric Informationcited by 39
  2. 2025Top-Down Semantic Refinement for Image Captioningcited by 16
  3. 2025Chimera: Diagnosing Shortcut Learning in Visual-Language Understandingcited by 7
  4. 2025Can Vision-Language Models Count? A Synthetic Benchmark and Analysis of Attention-Based Interventionscited by 6
  5. 2026From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Modelscited by 6
  6. 2026Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?cited by 6
  7. 2026Read the Paper, Write the Code: Agentic Reproduction of Social-Science Resultscited by 5
  8. 2025Compose and Fuse: Revisiting the Foundational Bottlenecks in Multimodal Reasoningcited by 4
  9. 2026CARE: Towards Clinical Accountability in Multi-Modal Medical Reasoning with an Evidence-Grounded Agentic Frameworkcited by 4
  10. 2026MJ1: Multimodal Judgment via Grounded Verificationcited by 3
  11. 2025Overcoming Vision Language Model Challenges in Diagram Understanding: A Proof-of-Concept with XML-Driven Large Language Models Solutionscited by 2
  12. 2025WIP: Large Language Model-Enhanced Smart Tutor for Undergraduate Circuit Analysiscited by 2
20 more
  1. 2026VOLMO: Versatile and Open Large Models for Ophthalmologycited by 2
  2. 2025ViStruct: Simulating Expert-Like Reasoning Through Task Decomposition and Visual Attention Cuescited by 1
  3. 2025Robust Diagram Reasoning: A Framework for Enhancing LVLM Performance on Visually Perturbed Scientific Diagramscited by 1
  4. 2026Multimodal Information Fusion for Chart Understanding: A Survey of MLLMs -- Evolution, Limitations, and Cognitive Enhancementcited by 1
  5. 2026Nodes Are Early, Edges Are Late: Probing Diagram Representations in Large Vision-Language Modelscited by 1
  6. 2025Ascending the Infinite Ladder: Benchmarking Spatial Deformation Reasoning in Vision-Language Modelscited by 0
  7. 2025Examining Vision Language Models through Multi-dimensional Experiments with Vision and Text Featurescited by 0
  8. 2025Seeing isn't Hearing: Benchmarking Vision Language Models at Interpreting Spectrogramscited by 0
  9. 2025Moving Pictures of Thought: Extracting Visual Knowledge in Charles S. Peirce's Manuscripts with Vision-Language Modelscited by 0
  10. 2025Action Without Interaction: Probing the Physical Foundations of Video LMMs via Contact-Release Detectioncited by 0
  11. 2026Beyond Pixels: Vector-to-Graph Transformation for Reliable Schematic Auditingcited by 0
  12. 2026An Online Reference-Free Evaluation Framework for Flowchart Image-to-Code Generationcited by 0
  13. 2026VLMs Trace Without Tracking: Diagnosing Failures in Visual Path Followingcited by 0
  14. 2026Chartographer: Counterfactual Chart Generation for Evaluating Vision-Language Modelscited by 0
  15. 2026Prior Bias in Vision Language Models on UML Diagram Interpretationcited by 0
  16. 2026ERUnderstand: Evaluating Vision-Language Models on Structured ER Diagramscited by 0
  17. 2026Investigating Relational Reasoning in VLMscited by 0
  18. 2026When Words Speak Louder than Images: Towards Understanding Language Bias in Vision-Language Modelscited by 0
  19. 2026Knossos and Ariadne: Benchmarking and Learning Complete Diagram Topology Extraction with Vision-Language Modelscited by 0
  20. 2026When to Rethink: Learning Multi-Perspective Self-Verification for Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Math Blind: Failures in Diagram Understanding Undermine Reasoning in MLLMs
  2. 2025Diagnosing Visual Reasoning: Challenges, Insights, and a Path Forward
  3. 2025Integrating Visual Interpretation and Linguistic Reasoning for Math Problem Solving
  4. 2025Structure-aware Contrastive Learning for Diagram Understanding of Multimodal Models
  5. 2025Oedipus and the Sphinx: Benchmarking and Improving Visual Language Models for Complex Graphic Reasoning
  6. 2026Do Diagrams Help Large Language Models Reason? Evidence from Syllogistic Reasoning
  7. 2024Can Large Language Models Understand Symbolic Graphics Programs?
  8. 2024HumanEval-V: Benchmarking High-Level Visual Reasoning with Complex Diagrams in Coding Tasks