mlsanityData to 14 Oct 2026

VisionGraph: Leveraging Large Multimodal Models for Graph Theory Problems in Visual Context

arXiv 2405.04950 ↗First posted 8 May 2024cs.CV cs.AI cs.CLcited by 31

On the map: LLM reasoning → Multimodal reasoning

also in 1 more
Abstract (arXiv)

Large Multimodal Models (LMMs) have achieved impressive success in visual understanding and reasoning, remarkably improving the performance of mathematical reasoning in a visual context. Yet, a challenging type of visual math lies in the multimodal graph theory problem, which demands that LMMs understand the graphical structures accurately and perform multi-step reasoning on the visual graph. Additionally, exploring multimodal graph theory problems will lead to more effective strategies in fields like biology, transportation, and robotics planning. To step forward in this direction, we are the first to design a benchmark named VisionGraph, used to explore the capabilities of advanced LMMs in solving multimodal graph theory problems. It encompasses eight complex graph problem tasks, from connectivity to shortest path problems. Subsequently, we present a Description-Program-Reasoning (DPR) chain to enhance the logical accuracy of reasoning processes through graphical structure description generation and algorithm-aware multi-step reasoning. Our extensive study shows that 1) GPT-4V outperforms Gemini Pro in multi-step graph reasoning; 2) All LMMs exhibit inferior perception accuracy for graphical structures, whether in zero/few-shot settings or with supervised fine-tuning (SFT), which further affects problem-solving performance; 3) DPR significantly improves the multi-step graph reasoning capabilities of LMMs and the GPT-4V (DPR) agent achieves SOTA performance.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 22 of its 29 references that are in the map. First posted on arXiv 8 May 2024.

Builds on

22 of 29 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts3 Oct

Other references · 21

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  4. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  5. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  6. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  7. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  8. 2023WizardMath: Empowering Mathematical Reasoning for Large Language Models via Reinforced Evol-Instructcited by 625
  9. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  10. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 286
  11. 2023Can Language Models Solve Graph Problems in Natural Language?cited by 252
  12. 2023MathPrompter: Mathematical Reasoning using Large Language Modelscited by 236
9 more
  1. 2022Vision-and-Language Navigation: A Survey of Tasks, Methods, and Future Directionscited by 117
  2. 2023On the Road with GPT-4V(ision): Early Explorations of Visual-Language Model on Autonomous Drivingcited by 80
  3. 2023GPT-4V(ision) for Robotics: Multimodal Task Planning from Human Demonstrationcited by 77
  4. 2023LLM-in-the-loop: Leveraging Large Language Model for Thematic Analysiscited by 70
  5. 2023A Systematic Evaluation of GPT-4V's Multimodal Capability for Medical Image Analysiscited by 37
  6. 2023LMEye: An Interactive Perception Network for Large Language Modelscited by 27
  7. 2023A Comprehensive Evaluation of GPT-4V on Knowledge-Intensive Visual Question Answeringcited by 23
  8. 2023WebVLN: Vision-and-Language Navigation on Websitescited by 19
  9. 2023AI for Mathematics: A Cognitive Science Perspectivecited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 10; 2025: 9; 2026: 12.

31 papers.

Sort:

By area:

  1. 2024Vision language models are blind: Failing to translate detailed visual features into wordscited by 147
  2. 2024GraphArena: Evaluating and Exploring Large Language Models on Graph Computationcited by 49
  3. 2024Can LLM Graph Reasoning Generalize beyond Pattern Memorization?cited by 47
  4. 2024Anim-Director: A Large Multimodal Model Powered Agent for Controllable Animation Video Generationcited by 37
  5. 2025Cross-Contrastive Clustering for Multimodal Attributed Graphs with Dual Graph Filteringcited by 14
  6. 2026Graph is a Substrate Across Data Modalitiescited by 14
  7. 2024GraCoRe: Benchmarking Graph Comprehension and Complex Reasoning in Large Language Modelscited by 13
  8. 2024GCoder: Improving Large Language Model for Generalized Graph Problem Solvingcited by 11
  9. 2024Revisiting the Graph Reasoning Ability of Large Language Models: Case Studies in Translation, Connectivity and Shortest Pathcited by 10
  10. 2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 9
  11. 2024GUNDAM: Aligning Large Language Models with Graph Understandingcited by 5
  12. 2025Graph-to-Vision: Multi-graph Understanding and Reasoning using Vision-Language Modelscited by 5
19 more
  1. 2025Directed-Tokens: A Robust Multi-Modality Alignment Approach to Large Language-Vision Modelscited by 5
  2. 2026DynamicGTR: Leveraging Graph Topology Representation Preferences to Boost VLM Capabilities on Graph QAscited by 4
  3. 2024VisGraphVar: A Benchmark Generator for Assessing Variability in Graph Analysis Using Large Vision-Language Modelscited by 3
  4. 2025Generalizable LLM Learning of Graph Synthetic Data with Post-training Alignmentcited by 3
  5. 2025Visual Graph Arena: Evaluating Visual Conceptualization of Vision and Multimodal Large Language Modelscited by 3
  6. 2026Exposing Weaknesses of Large Reasoning Models through Graph Algorithm Problemscited by 3
  7. 2024Multimodal LLMs Struggle with Basic Visual Network Analysis: a VNA Benchmarkcited by 2
  8. 2025See or Say Graphs: Agent-Driven Scalable Graph Structure Understanding with Vision-Language Modelscited by 2
  9. 2026Evaluating Relational Reasoning in LLMs with RELcited by 2
  10. 2026GraphVerse: A Comprehensive Visual Graph Reasoning Benchmark for Multimodal Large Language Modelscited by 2
  11. 2025Harnessing Adaptive Topology Representations for Zero-Shot Graph Question Answeringcited by 1
  12. 2026Feynman: Knowledge-Infused Diagramming Agent for Scalable Visual Designscited by 1
  13. 2025Structure-Aware Cooperative Ensemble Evolutionary Optimization on Combinatorial Problems with Multimodal Large Language Modelscited by 0
  14. 2026HyperGVL: Benchmarking and Improving Large Vision-Language Models in Hypergraph Understanding and Reasoningcited by 0
  15. 2026VLMs Trace Without Tracking: Diagnosing Failures in Visual Path Followingcited by 0
  16. 2026Prior Bias in Vision Language Models on UML Diagram Interpretationcited by 0
  17. 2026GTA: Graph Theory Agent and Benchmark for Algorithmic Graph Reasoning with LLMscited by 0
  18. 2026Knossos and Ariadne: Benchmarking and Learning Complete Diagram Topology Extraction with Vision-Language Modelscited by 0
  19. 2026MVGraph: Benchmarking LVLMs on Evidence Composition across Multi-View Visual Graphcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026When Vision Meets Graphs: A Survey on Graph Reasoning and Learning
  2. 2024Seeing the Forest and the Trees: Solving Visual Graph and Tree Based Data Structure Problems using Large Multimodal Models
  3. 2024Benchmarking and Improving Large Vision-Language Models for Fundamental Visual Graph Understanding and Reasoning
  4. 2026Mario: Multimodal Graph Reasoning with Large Language Models
  5. 2024GITA: Graph to Visual and Textual Integration for Vision-Language Graph Reasoning
  6. 2025Bridging Visualization and Optimization: Multimodal Large Language Models on Graph-Structured Combinatorial Optimization
  7. 2025Towards Understanding Graphical Perception in Large Multimodal Models
  8. 2026ViSR-KGC: Visual Subgraph Reasoning with Vision-Language Models for Multimodal Knowledge Graph Completion