mlsanityData to 14 Oct 2026

GRAPHGPT-O: Synergistic Multimodal Comprehension and Generation on Graphs

arXiv 2502.11925 ↗First posted 17 Feb 2025cs.AI cs.CV cs.LGcited by 17

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

The rapid development of Multimodal Large Language Models (MLLMs) has enabled the integration of multiple modalities, including texts and images, within the large language model (LLM) framework. However, texts and images are usually interconnected, forming a multimodal attributed graph (MMAG). It is underexplored how MLLMs can incorporate the relational information (textiti.e., graph structure) and semantic information (textiti.e., texts and images) on such graphs for multimodal comprehension and generation. In this paper, we propose GraphGPT-o, which supports omni-multimodal understanding and creation on MMAGs. We first comprehensively study linearization variants to transform semantic and structural information as input for MLLMs. Then, we propose a hierarchical aligner that enables deep graph encoding, bridging the gap between MMAGs and MLLMs. Finally, we explore the inference choices, adapting MLLM to interleaved text and image generation in graph scenarios. Extensive experiments on three datasets from different domains demonstrate the effectiveness of our proposed method. Datasets and codes will be open-sourced upon acceptance.

Key facts. Cited by 17 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 35 of its 43 references that are in the map. First posted on arXiv 17 Feb 2025.

Builds on

35 of 43 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Flamingo: a Visual Language Model for Few-Shot Learning29 Apr

Other references · 34

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2016Geometric deep learning: going beyond Euclidean datacited by 1,048
  8. 2018Predict then Propagate: Graph Neural Networks meet Personalized PageRankcited by 993
  9. 2024Chameleon: Mixed-Modal Early-Fusion Foundation Modelscited by 927
  10. 2023A Survey on Multimodal Large Language Modelscited by 909
  11. 2023Generative Multimodal Models are In-Context Learnerscited by 446
  12. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
22 more
  1. 2023GraphGPT: Graph Instruction Tuning for Large Language Modelscited by 226
  2. 2023Language is All a Graph Needscited by 220
  3. 2023Large Language Models on Graphs: A Comprehensive Surveycited by 192
  4. 2024GNN-RAG: Graph Neural Retrieval for Large Language Model Reasoningcited by 128
  5. 2024Graph Chain-of-Thought: Augmenting Large Language Models by Reasoning on Graphscited by 119
  6. 2023Art and the science of generative AI: A deeper divecited by 95
  7. 2024ZeroG: Investigating Cross-dataset Zero-shot Transferability in Graphscited by 85
  8. 2023Graph-ToolFormer: To Empower LLMs with Graph Reasoning Ability via Prompt Augmented by ChatGPTcited by 80
  9. 2024A Survey of Large Language Models for Graphscited by 72
  10. 2024A Review of Modern Recommender Systems Using Generative Models (Gen-RecSys)cited by 58
  11. 2024InstructGraph: Boosting Large Language Models via Graph-centric Instruction Tuning and Preference Alignmentcited by 56
  12. 2024Can we Soft Prompt LLMs for Graph Learning Tasks?cited by 52
  13. 2024GLBench: A Comprehensive Benchmark for Graph with Large Language Modelscited by 36
  14. 2024GraphWiz: An Instruction-Following Language Model for Graph Problemscited by 32
  15. 2022Draw Your Art Dream: Diverse Digital Art Synthesis with Multimodal Guided Diffusioncited by 30
  16. 2022Heterformer: Transformer-based Deep Node Representation Learning on Heterogeneous Text-Rich Networkscited by 24
  17. 2022EdgeFormer: A Parameter-Efficient Transformer for On-Device Seq2seq Generationcited by 21
  18. 2024GAugLLM: Improving Graph Contrastive Learning for Text-Attributed Graphs with Large Language Modelscited by 17
  19. 2024Multi-Behavior Generative Recommendationcited by 15
  20. 2024UniGLM: Training One Unified Language Model for Text-Attributed Graph Embeddingcited by 14
  21. 2024InstructG2I: Synthesizing Images from Multimodal Attributed Graphscited by 14
  22. 2024Investigating Instruction Tuning Large Language Models on Graphscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 16.

17 papers.

Sort:

By area:

  1. 2026OpenMAG: A Comprehensive Benchmark for Multimodal-Attributed Graphcited by 13
  2. 2026Toward Effective Multimodal Graph Foundation Model: A Divide-and-Conquer Based Approachcited by 7
  3. 2026TMTE: Effective Multimodal Graph Learning with Task-aware Modality and Topology Co-evolutioncited by 3
  4. 2026CAMPA: Efficient and Aligned Multimodal Graph Learning via Decoupled Propagation and Aggregationcited by 2
  5. 2026LION: A Clifford Neural Paradigm for Multimodal-Attributed Graph Learningcited by 2
  6. 2026PerfGuard: A Performance-Aware Agent for Visual Content Generationcited by 1
  7. 2026Context-aware Modality-Topology Co-Alignment for Multimodal Attributed Graphscited by 1
  8. 2025Towards Multimodal Graph Large Language Modelcited by 0
  9. 2026LION: A Clifford Neural Paradigm for Multimodal-Attributed Graph Learningcited by 0
  10. 2026OptiMAG: Structure-Semantic Alignment via Unbalanced Optimal Transportcited by 0
  11. 2026Stable Multimodal Graph Unlearning via Feature-Dimension Aware Quantile Selectioncited by 0
  12. 2026GOMA: Toward Structure-Driven Multimodal Alignment from a Graph Signal Smoothing Perspectivecited by 0
5 more
  1. 2026Multimodal Graph Negative Learningcited by 0
  2. 2026SMGFM: Spectral Multimodal Graph Pretraining for Multimodal-Attributed Graphscited by 0
  3. 2026Towards Unified Multimodal Graph Foundation Model: A Bridge-Router-Adapter Based Approachcited by 0
  4. 2026GraphSelect for Budgeted Representation Selection in Multimodal Graph Inferencecited by 0
  5. 2026Coupling Perception and Reasoning in Federated Multimodal Graph Foundation Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MLaGA: Multimodal Large Language and Graph Assistant
  2. 2025Graph-MLLM: Harnessing Multimodal Large Language Models for Multimodal Graph Learning
  3. 2024Exploring Graph Structure Comprehension Ability of Multimodal Large Language Models: Case Studies
  4. 2026Mario: Multimodal Graph Reasoning with Large Language Models
  5. 2026Scene Graph Thinking: Reinforcing Structured Visual Reasoning for Multimodal Large Language Models
  6. 2026GraphVLM: Benchmarking Vision Language Models for Multimodal Graph Learning
  7. 2023Which Modality should I use -- Text, Motif, or Image? : Understanding Graphs with Large Language Models
  8. 2026GraphVerse: A Comprehensive Visual Graph Reasoning Benchmark for Multimodal Large Language Models