mlsanityData to 7 Oct 2026

VisGraphVar: A Benchmark Generator for Assessing Variability in Graph Analysis Using Large Vision-Language Models

arXiv 2411.14832 ↗First posted 22 Nov 2024cs.CV cs.AI cs.CLcited by 2

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

The fast advancement of Large Vision-Language Models (LVLMs) has shown immense potential. These models are increasingly capable of tackling abstract visual tasks. Geometric structures, particularly graphs with their inherent flexibility and complexity, serve as an excellent benchmark for evaluating these models' predictive capabilities. While human observers can readily identify subtle visual details and perform accurate analyses, our investigation reveals that state-of-the-art LVLMs exhibit consistent limitations in specific visual graph scenarios, especially when confronted with stylistic variations. In response to these challenges, we introduce VisGraphVar (Visual Graph Variability), a customizable benchmark generator able to produce graph images for seven distinct task categories (detection, classification, segmentation, pattern recognition, link prediction, reasoning, matching), designed to systematically evaluate the strengths and limitations of individual LVLMs. We use VisGraphVar to produce 990 graph images and evaluate six LVLMs, employing two distinct prompting strategies, namely zero-shot and chain-of-thought. The findings demonstrate that variations in visual attributes of images (e.g., node labeling and layout) and the deliberate inclusion of visual imperfections, such as overlapping nodes, significantly affect model performance. This research emphasizes the importance of a comprehensive evaluation across graph-related tasks, extending beyond reasoning alone. VisGraphVar offers valuable insights to guide the development of more reliable and robust systems capable of performing advanced visual graph analysis.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 40 of its 57 references that are in the map. First posted on arXiv 22 Nov 2024.

Builds on

40 of 57 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Vision-Language Models Can Self-Improve Reasoning via Reflection30 Oct
  2. 2024Multimodal LLMs Struggle with Basic Visual Network Analysis: a VNA Benchmark10 May
  3. 2024VisionGraph: Leveraging Large Multimodal Models for Graph Theory Problems in Visual Context8 May
  4. 2023Vision-Language Models for Vision Tasks: A Survey3 Apr

Other references · 36

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  6. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  7. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  8. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  9. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  10. 2023A Survey on Multimodal Large Language Modelscited by 898
  11. 2018Link Prediction Based on Graph Neural Networkscited by 829
  12. 2021Florence: A New Foundation Model for Computer Visioncited by 802
24 more
  1. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 622
  2. 2022Towards Reasoning in Large Language Models: A Surveycited by 593
  3. 2024A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applicationscited by 428
  4. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 352
  5. 2023The False Promise of Imitating Proprietary LLMscited by 254
  6. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  7. 2019Graph Matching Networks for Learning the Similarity of Graph Structured Objectscited by 202
  8. 2024OmniMedVQA: A New Large-Scale Comprehensive Evaluation Benchmark for Medical LVLMcited by 182
  9. 2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoningcited by 158
  10. 2024Fine-Tuning Large Vision-Language Models as Decision-Making Agents via Reinforcement Learningcited by 158
  11. 2024Why are Visually-Grounded Language Models Bad at Image Classification?cited by 101
  12. 2024Benchmarking and Improving Detail Image Captioncited by 81
  13. 2023Evaluating Graph Neural Networks for Link Prediction: Current Pitfalls and New Benchmarkingcited by 69
  14. 2024A Survey of Prompt Engineering Methods in Large Language Models for Different NLP Taskscited by 46
  15. 2024LLMs Meet VLMs: Boost Open Vocabulary Object Detection with Fine-grained Descriptorscited by 39
  16. 2024GITA: Graph to Visual and Textual Integration for Vision-Language Graph Reasoningcited by 31
  17. 2024Are Large Vision Language Models up to the Challenge of Chart Comprehension and Reasoning? An Extensive Investigation into the Capabilities and Limitations of LVLMscited by 30
  18. 2023Revisiting Link Prediction: A Data Perspectivecited by 29
  19. 2024GPT-4o: Visual perception performance of multimodal large language models in piglet activity understandingcited by 12
  20. 2019Brain network similarity:Methods and applicationscited by 4
  21. 2020GraphPi: High Performance Graph Pattern Matching through Effective Redundancy Eliminationcited by 3
  22. 2022Attacking Shortest Paths by Cutting Edgescited by 3
  23. 2018Survey of Graph Analysis Applicationscited by 2
  24. 2024Graph matching based on similarities in structure and attributescited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Graph is a Substrate Across Data Modalitiescited by 13
  2. 2026When Vision Meets Graphs: A Survey on Graph Reasoning and Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Benchmarking and Improving Large Vision-Language Models for Fundamental Visual Graph Understanding and Reasoning
  2. 2026GraphVerse: A Comprehensive Visual Graph Reasoning Benchmark for Multimodal Large Language Models
  3. 2026Seeing Graphs Like Humans: Benchmarking Computational Measures and MLLMs for Similarity Assessment ↗
  4. 2025Graph-to-Vision: Multi-graph Understanding and Reasoning using Vision-Language Models
  5. 2026GraphVLM: Benchmarking Vision Language Models for Multimodal Graph Learning
  6. 2026One Model, Many Graphs: Learning over Attributed Graphs across Heterogeneous Modalities with Vision-Language Models
  7. 2025Can VLMs Assess Similarity Between Graph Visualizations?
  8. 2025Visual Graph Arena: Evaluating Visual Conceptualization of Vision and Multimodal Large Language Models