mlsanityData to 7 Oct 2026

GITA: Graph to Visual and Textual Integration for Vision-Language Graph Reasoning

arXiv 2402.02130 ↗First posted 3 Feb 2024cs.CLcited by 31

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Large Language Models (LLMs) are increasingly used for various tasks with graph structures. Though LLMs can process graph information in a textual format, they overlook the rich vision modality, which is an intuitive way for humans to comprehend structural information and conduct general graph reasoning. The potential benefits and capabilities of representing graph structures as visual images (i.e., textitvisual graph) are still unexplored. To fill the gap, we innovatively propose an end-to-end framework, called Graph to vIsual and Textual IntegrAtion (GITA), which firstly incorporates visual graphs into general graph reasoning. Besides, we establish Graph-based Vision-Language Question Answering (GVLQA) dataset from existing graph data, which is the first vision-language dataset for general graph reasoning purposes. Extensive experiments on the GVLQA dataset and five real-world datasets show that GITA outperforms mainstream LLMs in terms of general graph reasoning capabilities. Moreover, We highlight the effectiveness of the layout augmentation on visual graphs and pretraining on the GVLQA dataset.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 46 of its 72 references that are in the map. First posted on arXiv 3 Feb 2024.

Builds on

46 of 72 references.

References · 46

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  5. 2017Graph Attention Networkscited by 8,854
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2017Inductive Representation Learning on Large Graphscited by 6,284
  9. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  10. 2018How Powerful are Graph Neural Networks?cited by 4,355
  11. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  12. 2017Neural Message Passing for Quantum Chemistrycited by 3,460
34 more
  1. 2019A Comprehensive Survey on Graph Neural Networkscited by 2,906
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  3. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  4. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  5. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  6. 2018Relational inductive biases, deep learning, and graph networkscited by 1,341
  7. 2020LightGCN: Simplifying and Powering Graph Convolution Network for Recommendationcited by 1,134
  8. 2017Neural Collaborative Filteringcited by 1,108
  9. 2016Revisiting Semi-Supervised Learning with Graph Embeddingscited by 970
  10. 2023MetaMath: Bootstrap Your Own Mathematical Questions for Large Language Modelscited by 724
  11. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 553
  12. 2016Interaction Networks for Learning about Objects, Relations and Physicscited by 503
  13. 2021Rethinking Graph Transformers with Spectral Attentioncited by 468
  14. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  15. 2015Neural Module Networkscited by 375
  16. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  17. 2023Exploring the Potential of Large Language Models (LLMs) in Learning on Graphscited by 271
  18. 2023Can Language Models Solve Graph Problems in Natural Language?cited by 249
  19. 2019RAVEN: A Dataset for Relational and Analogical Visual rEasoNingcited by 225
  20. 2023GraphGPT: Graph Instruction Tuning for Large Language Modelscited by 223
  21. 2023Language is All a Graph Needscited by 219
  22. 2023GPT4Graph: Can Large Language Models Understand Graph Structured Data ? An Empirical Evaluation and Benchmarkingcited by 209
  23. 2022Structure-Aware Transformer for Graph Representation Learningcited by 191
  24. 2020Graph-Bert: Only Attention is Needed for Learning Graph Representationscited by 176
  25. 2023Talk like a Graph: Encoding Graphs for Large Language Modelscited by 175
  26. 2023Harnessing Explanations: LLM-to-LM Interpreter for Enhanced Text-Attributed Graph Representation Learningcited by 132
  27. 2021Learning Intents behind Interactions with Knowledge Graph for Recommendationcited by 94
  28. 2021ConE: Cone Embeddings for Multi-Hop Reasoning over Knowledge Graphscited by 78
  29. 2021Knowledge-aware Coupled Graph Neural Network for Social Recommendationcited by 49
  30. 2022Mask and Reason: Pre-Training Knowledge Graph Transformers for Complex Logical Queriescited by 39
  31. 2023Forward-Backward Reasoning in Large Language Models for Mathematical Verificationcited by 39
  32. 2023Evaluating Large Language Models on Graphs: Performance Insights and Comparative Analysiscited by 35
  33. 2019Popularity Prediction on Social Platforms with Coupled Graph Neural Networkscited by 26
  34. 2023Effective Structured Prompting by Meta-Learning and Representative Verbalizercited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 14; 2026: 11.

31 papers.

Sort:

By area:

  1. 2024RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Modelscited by 84
  2. 2024GraphArena: Evaluating and Exploring Large Language Models on Graph Computationcited by 48
  3. 2024Mosaic of Modalities: A Comprehensive Benchmark for Multimodal Graph Learningcited by 28
  4. 2025Safe Delta: Consistently Preserving Safety when Fine-Tuning LLMs on Diverse Datasetscited by 17
  5. 2024Can Graph Descriptive Order Affect Solving Graph Problems with LLMs?cited by 16
  6. 2026Graph is a Substrate Across Data Modalitiescited by 13
  7. 2025GraphThought: Graph Combinatorial Optimization with Thought Generationcited by 9
  8. 2025ContextNav: Towards Agentic Multimodal In-Context Learningcited by 9
  9. 2025Model Generalization on Text Attribute Graphs: Principles with Large Language Modelscited by 8
  10. 2025MetaDefense: Defending Finetuning-based Jailbreak Attack Before and During Generationcited by 8
  11. 2025Large Language Models Meet Text-Attributed Graphs: A Survey of Integration Frameworks and Applicationscited by 5
  12. 2025NAUTILUS: A Large Multimodal Model for Underwater Scene Understandingcited by 5
19 more
  1. 2026DynamicGTR: Leveraging Graph Topology Representation Preferences to Boost VLM Capabilities on Graph QAscited by 4
  2. 2024VisGraphVar: A Benchmark Generator for Assessing Variability in Graph Analysis Using Large Vision-Language Modelscited by 2
  3. 2024CARL-GT: Evaluating Causal Reasoning Capabilities of Large Language Modelscited by 2
  4. 2025A Survey of Cross-domain Graph Learning: Progress and Future Directionscited by 2
  5. 2025Graph Drawing for LLMs: An Empirical Evaluationcited by 2
  6. 2025Open Your Eyes: Vision Enhances Message Passing Neural Networks in Link Predictioncited by 2
  7. 2025See or Say Graphs: Agent-Driven Scalable Graph Structure Understanding with Vision-Language Modelscited by 2
  8. 2025SPHINX: A Synthetic Environment for Visual Perception and Reasoningcited by 2
  9. 2025Harnessing Adaptive Topology Representations for Zero-Shot Graph Question Answeringcited by 1
  10. 2026FARE: Fast-Slow Agentic Robotic Explorationcited by 1
  11. 2026Graph2Video: Leveraging Video Models to Model Dynamic Graph Evolutioncited by 1
  12. 2025Structure-Aware Cooperative Ensemble Evolutionary Optimization on Combinatorial Problems with Multimodal Large Language Modelscited by 0
  13. 2026When 2D Tasks Meet 1D Serialization: On Serialization Friction in Structured Taskscited by 0
  14. 2026Stable Multimodal Graph Unlearning via Feature-Dimension Aware Quantile Selectioncited by 0
  15. 2026MetaMoE: Diversity-Aware Proxy Selection for Privacy-Preserving Mixture-of-Experts Unificationcited by 0
  16. 2026SPARD: Defending Harmful Fine-Tuning Attack via Safety Projection with Relevance-Diversity Data Selectioncited by 0
  17. 2026Are Large Language Models Suitable for Graph Computation? Progress and Prospectscited by 0
  18. 2026RepoAtlas: Guiding Coding Agents via Evolving Multimodal Repository Viewscited by 0
  19. 2026VisKG-LM: Compiling Knowledge Graphs into Visual Memory for Multiple-Choice Question Answeringcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026GraphVerse: A Comprehensive Visual Graph Reasoning Benchmark for Multimodal Large Language Models
  2. 2024Benchmarking and Improving Large Vision-Language Models for Fundamental Visual Graph Understanding and Reasoning
  3. 2025Graph-to-Vision: Multi-graph Understanding and Reasoning using Vision-Language Models
  4. 2024VisionGraph: Leveraging Large Multimodal Models for Graph Theory Problems in Visual Context
  5. 2025Diagnosing Visual Reasoning: Challenges, Insights, and a Path Forward
  6. 2023Which Modality should I use -- Text, Motif, or Image? : Understanding Graphs with Large Language Models
  7. 2025VisReason: A Large-Scale Dataset for Visual Chain-of-Thought Reasoning
  8. 2025Visual Graph Arena: Evaluating Visual Conceptualization of Vision and Multimodal Large Language Models