# Do Vision-Language Models Really Understand Visual Language?

arXiv: https://arxiv.org/abs/2410.00193 · HTML: https://mlsanity.com/paper/2410.00193/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 34 of its 54 references that are in the map.
- First posted on arXiv 30 Sep 2024.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 30 of 34)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering](https://mlsanity.com/paper/2209.09513/) (2022)
- [Evaluating Object Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2305.10355/) (2023)
- [MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models](https://mlsanity.com/paper/2306.13394/) (2023)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities](https://mlsanity.com/paper/2308.02490/) (2023)
- [A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivity](https://mlsanity.com/paper/2302.04023/) (2023)
- [A Diagram Is Worth A Dozen Images](https://mlsanity.com/paper/1603.07396/) (2016)
- [MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?](https://mlsanity.com/paper/2403.14624/) (2024)
- [Visual Programming: Compositional visual reasoning without training](https://mlsanity.com/paper/2211.11559/) (2022)
- [Is ChatGPT a General-Purpose Natural Language Processing Task Solver?](https://mlsanity.com/paper/2302.06476/) (2023)
- [AMBER: An LLM-free Multi-dimensional Benchmark for MLLMs Hallucination Evaluation](https://mlsanity.com/paper/2311.07397/) (2023)
- [IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoning](https://mlsanity.com/paper/2110.13214/) (2021)
- [Talk like a Graph: Encoding Graphs for Large Language Models](https://mlsanity.com/paper/2310.04560/) (2023)
- [Causal Parrots: Large Language Models May Talk Causality But Are Not Causal](https://mlsanity.com/paper/2308.13067/) (2023)
- [Yin and Yang: Balancing and Answering Binary Visual Questions](https://mlsanity.com/paper/1511.05099/) (2015)
- [Evaluation and Analysis of Hallucination in Large Vision-Language Models](https://mlsanity.com/paper/2308.15126/) (2023)
- [Adversarial VQA: A New Benchmark for Evaluating the Robustness of VQA Models](https://mlsanity.com/paper/2106.00245/) (2021)
- [Large Language Models Can be Lazy Learners: Analyze Shortcuts in In-Context Learning](https://mlsanity.com/paper/2305.17256/) (2023)
- [Large Language Models are Visual Reasoning Coordinators](https://mlsanity.com/paper/2310.15166/) (2023)
- [ShapeWorld - A new test methodology for multimodal language understanding](https://mlsanity.com/paper/1704.04517/) (2017)
- [Are Large Vision Language Models up to the Challenge of Chart Comprehension and Reasoning? An Extensive Investigation into the Capabilities and Limitations of LVLMs](https://mlsanity.com/paper/2406.00257/) (2024)
- [Doubly Right Object Recognition: A Why Prompt for Visual Rationales](https://mlsanity.com/paper/2212.06202/) (2022)
- [FlowVQA: Mapping Multimodal Logic in Visual Question Answering with Flowcharts](https://mlsanity.com/paper/2406.19237/) (2024)
- [FlowLearn: Evaluating Large Vision-Language Models on Flowchart Understanding](https://mlsanity.com/paper/2407.05183/) (2024)

## Built on by (most cited first, 30 of 32)

- [VisOnlyQA: Large Vision Language Models Still Struggle with Visual Perception of Geometric Information](https://mlsanity.com/paper/2412.00947/) (2024)
- [Top-Down Semantic Refinement for Image Captioning](https://mlsanity.com/paper/2510.22391/) (2025)
- [Chimera: Diagnosing Shortcut Learning in Visual-Language Understanding](https://mlsanity.com/paper/2509.22437/) (2025)
- [Can Vision-Language Models Count? A Synthetic Benchmark and Analysis of Attention-Based Interventions](https://mlsanity.com/paper/2511.17722/) (2025)
- [From Seeing to Thinking: Decoupling Perception and Reasoning Improves Post-Training of Vision-Language Models](https://mlsanity.com/paper/2605.20177/) (2026)
- [Seeing without Looking: Do Vision-Language Benchmarks Really Test Vision?](https://mlsanity.com/paper/2605.22903/) (2026)
- [Read the Paper, Write the Code: Agentic Reproduction of Social-Science Results](https://mlsanity.com/paper/2604.21965/) (2026)
- [Compose and Fuse: Revisiting the Foundational Bottlenecks in Multimodal Reasoning](https://mlsanity.com/paper/2509.23744/) (2025)
- [CARE: Towards Clinical Accountability in Multi-Modal Medical Reasoning with an Evidence-Grounded Agentic Framework](https://mlsanity.com/paper/2603.01607/) (2026)
- [MJ1: Multimodal Judgment via Grounded Verification](https://mlsanity.com/paper/2603.07990/) (2026)
- [Overcoming Vision Language Model Challenges in Diagram Understanding: A Proof-of-Concept with XML-Driven Large Language Models Solutions](https://mlsanity.com/paper/2502.04389/) (2025)
- [WIP: Large Language Model-Enhanced Smart Tutor for Undergraduate Circuit Analysis](https://mlsanity.com/paper/2506.08962/) (2025)
- [VOLMO: Versatile and Open Large Models for Ophthalmology](https://mlsanity.com/paper/2603.23953/) (2026)
- [ViStruct: Simulating Expert-Like Reasoning Through Task Decomposition and Visual Attention Cues](https://mlsanity.com/paper/2506.21762/) (2025)
- [Robust Diagram Reasoning: A Framework for Enhancing LVLM Performance on Visually Perturbed Scientific Diagrams](https://mlsanity.com/paper/2508.16972/) (2025)
- [Multimodal Information Fusion for Chart Understanding: A Survey of MLLMs -- Evolution, Limitations, and Cognitive Enhancement](https://mlsanity.com/paper/2602.10138/) (2026)
- [Nodes Are Early, Edges Are Late: Probing Diagram Representations in Large Vision-Language Models](https://mlsanity.com/paper/2603.02865/) (2026)
- [Ascending the Infinite Ladder: Benchmarking Spatial Deformation Reasoning in Vision-Language Models](https://mlsanity.com/paper/2507.02978/) (2025)
- [Examining Vision Language Models through Multi-dimensional Experiments with Vision and Text Features](https://mlsanity.com/paper/2509.08266/) (2025)
- [Seeing isn't Hearing: Benchmarking Vision Language Models at Interpreting Spectrograms](https://mlsanity.com/paper/2511.13225/) (2025)
- [Moving Pictures of Thought: Extracting Visual Knowledge in Charles S. Peirce's Manuscripts with Vision-Language Models](https://mlsanity.com/paper/2511.13378/) (2025)
- [Action Without Interaction: Probing the Physical Foundations of Video LMMs via Contact-Release Detection](https://mlsanity.com/paper/2511.20162/) (2025)
- [Beyond Pixels: Vector-to-Graph Transformation for Reliable Schematic Auditing](https://mlsanity.com/paper/2602.11678/) (2026)
- [An Online Reference-Free Evaluation Framework for Flowchart Image-to-Code Generation](https://mlsanity.com/paper/2602.13376/) (2026)
- [VLMs Trace Without Tracking: Diagnosing Failures in Visual Path Following](https://mlsanity.com/paper/2605.15672/) (2026)
- [Chartographer: Counterfactual Chart Generation for Evaluating Vision-Language Models](https://mlsanity.com/paper/2605.27311/) (2026)
- [Prior Bias in Vision Language Models on UML Diagram Interpretation](https://mlsanity.com/paper/2607.02853/) (2026)
- [ERUnderstand: Evaluating Vision-Language Models on Structured ER Diagrams](https://mlsanity.com/paper/2607.24707/) (2026)
- [Investigating Relational Reasoning in VLMs](https://mlsanity.com/paper/2608.23518/) (2026)
- [When Words Speak Louder than Images: Towards Understanding Language Bias in Vision-Language Models](https://mlsanity.com/paper/2609.35272/) (2026)

## Abstract (arXiv)

Visual language is a system of communication that conveys information through symbols, shapes, and spatial arrangements. Diagrams are a typical example of a visual language depicting complex concepts and their relationships in the form of an image. The symbolic nature of diagrams presents significant challenges for building models capable of understanding them. Recent studies suggest that Large Vision-Language Models (LVLMs) can even tackle complex reasoning tasks involving diagrams. In this paper, we investigate this phenomenon by developing a comprehensive test suite to evaluate the diagram comprehension capability of LVLMs. Our test suite uses a variety of questions focused on concept entities and their relationships over a set of synthetic as well as real diagrams across domains to evaluate the recognition and reasoning abilities of models. Our evaluation of LVLMs shows that while they can accurately identify and reason about entities, their ability to understand relationships is notably limited. Further testing reveals that the decent performance on diagram understanding largely stems from leveraging their background knowledge as shortcuts to identify and reason about the relational information. Thus, we conclude that LVLMs have a limited capability for genuine diagram understanding, and their impressive performance in diagram reasoning is an illusion emanating from other confounding factors, such as the background knowledge in the models.