# When Graph meets Multimodal: Benchmarking and Meditating on Multimodal Attributed Graphs Learning

arXiv: https://arxiv.org/abs/2410.09132 · HTML: https://mlsanity.com/paper/2410.09132/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 43 references that are in the map.
- First posted on arXiv 11 Oct 2024.

## Builds on (most cited first, 24 of 24)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [Open Graph Benchmark: Datasets for Machine Learning on Graphs](https://mlsanity.com/paper/2005.00687/) (2020)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [Pitfalls of Graph Neural Network Evaluation](https://mlsanity.com/paper/1811.05868/) (2018)
- [ConvNeXt V2: Co-designing and Scaling ConvNets with Masked Autoencoders](https://mlsanity.com/paper/2301.00808/) (2023)
- [GraphGPT: Graph Instruction Tuning for Large Language Models](https://mlsanity.com/paper/2310.13023/) (2023)
- [RedCaps: web-curated image-text data created by the people, for the people](https://mlsanity.com/paper/2111.11431/) (2021)
- [Training Graph Neural Networks with 1000 Layers](https://mlsanity.com/paper/2106.07476/) (2021)
- [Learning on Large-scale Text-attributed Graphs via Variational Inference](https://mlsanity.com/paper/2210.14709/) (2022)
- [SimTeG: A Frustratingly Simple Approach Improves Textual Graph Learning](https://mlsanity.com/paper/2308.02565/) (2023)
- [A Large-Scale Database for Graph Representation Learning](https://mlsanity.com/paper/2011.07682/) (2020)
- [Graph Robustness Benchmark: Benchmarking the Adversarial Robustness of Graph Machine Learning](https://mlsanity.com/paper/2111.04314/) (2021)
- [DTGB: A Comprehensive Benchmark for Dynamic Text-Attributed Graphs](https://mlsanity.com/paper/2406.12072/) (2024)
- [Prompt-based Node Feature Extractor for Few-shot Learning on Text-Attributed Graphs](https://mlsanity.com/paper/2309.02848/) (2023)
- [TEG-DB: A Comprehensive Dataset and Benchmark of Textual-Edge Graphs](https://mlsanity.com/paper/2406.10310/) (2024)
- [Unleash LLMs Potential for Sequential Recommendation by Coordinating Dual Dynamic Index Mechanism](https://mlsanity.com/paper/2409.09253/) (2024)

## Built on by (most cited first, 22 of 22)

- [MLaGA: Multimodal Large Language and Graph Assistant](https://mlsanity.com/paper/2506.02568/) (2025)
- [Toward Effective Multimodal Graph Foundation Model: A Divide-and-Conquer Based Approach](https://mlsanity.com/paper/2602.04116/) (2026)
- [AgentGL: Towards Agentic Graph Learning with LLMs via Reinforcement Learning](https://mlsanity.com/paper/2604.05846/) (2026)
- [NTSFormer: A Self-Teaching Graph Transformer for Multimodal Isolated Cold-Start Node Classification](https://mlsanity.com/paper/2507.04870/) (2025)
- [Mario: Multimodal Graph Reasoning with Large Language Models](https://mlsanity.com/paper/2603.05181/) (2026)
- [TMTE: Effective Multimodal Graph Learning with Task-aware Modality and Topology Co-evolution](https://mlsanity.com/paper/2603.27723/) (2026)
- [PRISM: Topology-Aware Cross-Modal Imputation for Modality-Deficient Federated Graph Learning](https://mlsanity.com/paper/2606.09301/) (2026)
- [CAMPA: Efficient and Aligned Multimodal Graph Learning via Decoupled Propagation and Aggregation](https://mlsanity.com/paper/2605.11468/) (2026)
- [LION: A Clifford Neural Paradigm for Multimodal-Attributed Graph Learning](https://mlsanity.com/paper/2608.24795/) (2026)
- [Modality as Heterogeneity: Node Splitting and Graph Rewiring for Multimodal Graph Learning](https://mlsanity.com/paper/2602.00067/) (2026)
- [Rel-MOSS: Towards Imbalanced Relational Deep Learning on Relational Databases](https://mlsanity.com/paper/2603.07916/) (2026)
- [One Model, Many Graphs: Learning over Attributed Graphs across Heterogeneous Modalities with Vision-Language Models](https://mlsanity.com/paper/2607.19128/) (2026)
- [LION: A Clifford Neural Paradigm for Multimodal-Attributed Graph Learning](https://mlsanity.com/paper/2601.21453/) (2026)
- [Beyond the Aggregation Dilemma: Prior-Retaining Decoupled Learning for Multimodal Graphs](https://mlsanity.com/paper/2605.24684/) (2026)
- [Multimodal Graph Negative Learning](https://mlsanity.com/paper/2606.12863/) (2026)
- [SMGFM: Spectral Multimodal Graph Pretraining for Multimodal-Attributed Graphs](https://mlsanity.com/paper/2606.12867/) (2026)
- [UNIT: Unleash Large Language Models Potential for Graph Continual Learning](https://mlsanity.com/paper/2607.10159/) (2026)
- [ProTAGAD: A Foundation Model for TAG Anomaly Detection with Decoupled Topological and Textual Prototypes](https://mlsanity.com/paper/2608.10699/) (2026)
- [ReCoG: Reciprocal Co-Evolution for Multimodal Graph Learning](https://mlsanity.com/paper/2608.22786/) (2026)
- [GraphSelect for Budgeted Representation Selection in Multimodal Graph Inference](https://mlsanity.com/paper/2609.33561/) (2026)
- [GraphMAS: A Systematic Benchmark of Multi-Agent Coordination for Graph Learning](https://mlsanity.com/paper/2609.39777/) (2026)
- [No-Free-Graph: Learning When Multimodal Data Should Be Graphified](https://mlsanity.com/paper/2610.02768/) (2026)

## Abstract (arXiv)

Multimodal Attributed Graphs (MAGs) are ubiquitous in real-world applications, encompassing extensive knowledge through multimodal attributes attached to nodes (e.g., texts and images) and topological structure representing node interactions. Despite its potential to advance diverse research fields like social networks and e-commerce, MAG representation learning (MAGRL) remains underexplored due to the lack of standardized datasets and evaluation frameworks. In this paper, we first propose MAGB, a comprehensive MAG benchmark dataset, featuring curated graphs from various domains with both textual and visual attributes. Based on MAGB dataset, we further systematically evaluate two mainstream MAGRL paradigms: textitGNN-as-Predictor, which integrates multimodal attributes via Graph Neural Networks (GNNs), and textitVLM-as-Predictor, which harnesses Vision Language Models (VLMs) for zero-shot reasoning. Extensive experiments on MAGB reveal following critical insights: textit(i) Modality significances fluctuate drastically with specific domain characteristics. textit(ii) Multimodal embeddings can elevate the performance ceiling of GNNs. However, intrinsic biases among modalities may impede effective training, particularly in low-data scenarios. textit(iii) VLMs are highly effective at generating multimodal embeddings that alleviate the imbalance between textual and visual attributes. These discoveries, which illuminate the synergy between multimodal attributes and graph topologies, contribute to reliable benchmarks, paving the way for future MAG research. The MAGB dataset and evaluation pipeline are publicly available at https://github.com/sktsherlock/MAGB.