# Benchmarking and Improving Large Vision-Language Models for Fundamental Visual Graph Understanding and Reasoning

arXiv: https://arxiv.org/abs/2412.13540 · HTML: https://mlsanity.com/paper/2412.13540/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 13 of its 50 references that are in the map.
- First posted on arXiv 18 Dec 2024.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 13 of 13)

- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [TUDataset: A collection of benchmark datasets for learning with graphs](https://mlsanity.com/paper/2007.08663/) (2020)
- [MiniGPT-v2: large language model as a unified interface for vision-language multi-task learning](https://mlsanity.com/paper/2310.09478/) (2023)
- [LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models](https://mlsanity.com/paper/2407.07895/) (2024)
- [mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Models](https://mlsanity.com/paper/2408.04840/) (2024)
- [SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Models](https://mlsanity.com/paper/2311.07575/) (2023)
- [Wiki-CS: A Wikipedia-Based Benchmark for Graph Neural Networks](https://mlsanity.com/paper/2007.02901/) (2020)
- [InternLM-XComposer-2.5: A Versatile Large Vision Language Model Supporting Long-Contextual Input and Output](https://mlsanity.com/paper/2407.03320/) (2024)
- [Test of Time: A Benchmark for Evaluating LLMs on Temporal Reasoning](https://mlsanity.com/paper/2406.09170/) (2024)
- [GraphArena: Evaluating and Exploring Large Language Models on Graph Computation](https://mlsanity.com/paper/2407.00379/) (2024)
- [Large Language Models Are Cross-Lingual Knowledge-Free Reasoners](https://mlsanity.com/paper/2406.16655/) (2024)

## Built on by (most cited first, 19 of 19)

- [Evaluating and Steering Modality Preferences in Multimodal Large Language Model](https://mlsanity.com/paper/2505.20977/) (2025)
- [Why Reasoning Matters? A Survey of Advancements in Multimodal Reasoning (v1)](https://mlsanity.com/paper/2504.03151/) (2025)
- [Thinking in Character: Advancing Role-Playing Agents with Role-Aware Reasoning](https://mlsanity.com/paper/2506.01748/) (2025)
- [Graph is a Substrate Across Data Modalities](https://mlsanity.com/paper/2601.22384/) (2026)
- [Memory-augmented Query Reconstruction for LLM-based Knowledge Graph Reasoning](https://mlsanity.com/paper/2503.05193/) (2025)
- [LoCoT2V-Bench: Benchmarking Long-Form and Complex Text-to-Video Generation](https://mlsanity.com/paper/2510.26412/) (2025)
- [Instruction Anchor: Dissecting the Mechanistic Dynamics of Modality Arbitration](https://mlsanity.com/paper/2602.03677/) (2026)
- [Beyond the Vision Encoder: Identifying and Mitigating Spatial Bias in Large Vision-Language Models](https://mlsanity.com/paper/2509.21984/) (2025)
- [See or Say Graphs: Agent-Driven Scalable Graph Structure Understanding with Vision-Language Models](https://mlsanity.com/paper/2510.16769/) (2025)
- [Nodes Are Early, Edges Are Late: Probing Diagram Representations in Large Vision-Language Models](https://mlsanity.com/paper/2603.02865/) (2026)
- [PPU-Bench:Real World Benchmark for Personalized Partial Unlearning in Vision Language Models](https://mlsanity.com/paper/2605.08800/) (2026)
- [Beyond Token-Level Policy Gradients for Complex Reasoning with Large Language Models](https://mlsanity.com/paper/2602.14386/) (2026)
- [Are Large Language Models Suitable for Graph Computation? Progress and Prospects](https://mlsanity.com/paper/2606.06865/) (2026)
- [Prior Bias in Vision Language Models on UML Diagram Interpretation](https://mlsanity.com/paper/2607.02853/) (2026)
- [A Systematic Survey on Image Description Techniques for STEM Domains](https://mlsanity.com/paper/2607.21611/) (2026)
- [RepoAtlas: Guiding Coding Agents via Evolving Multimodal Repository Views](https://mlsanity.com/paper/2609.16936/) (2026)
- [VisKG-LM: Compiling Knowledge Graphs into Visual Memory for Multiple-Choice Question Answering](https://mlsanity.com/paper/2609.19158/) (2026)
- [Knossos and Ariadne: Benchmarking and Learning Complete Diagram Topology Extraction with Vision-Language Models](https://mlsanity.com/paper/2610.04721/) (2026)
- [MVGraph: Benchmarking LVLMs on Evidence Composition across Multi-View Visual Graph](https://mlsanity.com/paper/2610.12961/) (2026)

## Abstract (arXiv)

Large Vision-Language Models (LVLMs) have demonstrated remarkable performance across diverse tasks. Despite great success, recent studies show that LVLMs encounter substantial limitations when engaging with visual graphs. To study the reason behind these limitations, we propose VGCure, a comprehensive benchmark covering 22 tasks for examining the fundamental graph understanding and reasoning capacities of LVLMs. Extensive evaluations conducted on 14 LVLMs reveal that LVLMs are weak in basic graph understanding and reasoning tasks, particularly those concerning relational or structurally complex information. Based on this observation, we propose a structure-aware fine-tuning framework to enhance LVLMs with structure learning abilities through three self-supervised learning tasks. Experiments validate the effectiveness of our method in improving LVLMs' performance on fundamental and downstream graph learning tasks, as well as enhancing their robustness against complex visual graphs.