# Text-space Graph Foundation Models: Comprehensive Benchmarks and New Insights

arXiv: https://arxiv.org/abs/2406.10727 · HTML: https://mlsanity.com/paper/2406.10727/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 73 of its 96 references that are in the map.
- First posted on arXiv 15 Jun 2024.

## Builds on (most cited first, 30 of 73)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Emergent Abilities of Large Language Models](https://mlsanity.com/paper/2206.07682/) (2022)
- [Open Graph Benchmark: Datasets for Machine Learning on Graphs](https://mlsanity.com/paper/2005.00687/) (2020)
- [Simplifying Graph Convolutional Networks](https://mlsanity.com/paper/1902.07153/) (2019)
- [Graph Convolutional Neural Networks for Web-Scale Recommender Systems](https://mlsanity.com/paper/1806.01973/) (2018)
- [Deep Graph Infomax](https://mlsanity.com/paper/1809.10341/) (2018)
- [Revisiting Semi-Supervised Learning with Graph Embeddings](https://mlsanity.com/paper/1603.08861/) (2016)
- [Graph Neural Networks for Social Recommendation](https://mlsanity.com/paper/1902.07243/) (2019)
- [GraphSAINT: Graph Sampling Based Inductive Learning Method](https://mlsanity.com/paper/1907.04931/) (2019)
- [GCC: Graph Contrastive Coding for Graph Neural Network Pre-Training](https://mlsanity.com/paper/2006.09963/) (2020)
- [GraphMAE: Self-Supervised Masked Graph Autoencoders](https://mlsanity.com/paper/2205.10803/) (2022)
- [A critical look at the evaluation of GNNs under heterophily: Are we really making progress?](https://mlsanity.com/paper/2302.11640/) (2023)
- [Pre-Trained Models: Past, Present and Future](https://mlsanity.com/paper/2106.07139/) (2021)
- [Graph Self-Supervised Learning: A Survey](https://mlsanity.com/paper/2103.00111/) (2021)
- [Exploring the Potential of Large Language Models (LLMs) in Learning on Graphs](https://mlsanity.com/paper/2307.03393/) (2023)
- [One for All: Towards Training One Graph Model for All Classification Tasks](https://mlsanity.com/paper/2310.00149/) (2023)
- [GraphGPT: Graph Instruction Tuning for Large Language Models](https://mlsanity.com/paper/2310.13023/) (2023)
- [G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering](https://mlsanity.com/paper/2402.07630/) (2024)
- [LinkBERT: Pretraining Language Models with Document Links](https://mlsanity.com/paper/2203.15827/) (2022)
- [Large Language Models on Graphs: A Comprehensive Survey](https://mlsanity.com/paper/2312.02783/) (2023)

## Built on by (most cited first, 30 of 44)

- [Position: Graph Learning Will Lose Relevance Due To Poor Benchmarks](https://mlsanity.com/paper/2502.14546/) (2025)
- [Towards Graph Foundation Models: Learning Generalities Across Graphs via Task-Trees](https://mlsanity.com/paper/2412.16441/) (2024)
- [LLM as GNN: Graph Vocabulary Learning for Text-Attributed Graph Foundation Models](https://mlsanity.com/paper/2503.03313/) (2025)
- [GRAVER: Generative Graph Vocabularies for Robust Graph Foundation Models Fine-tuning](https://mlsanity.com/paper/2511.05592/) (2025)
- [One Model for One Graph: A New Perspective for Pretraining with Cross-domain Graphs](https://mlsanity.com/paper/2412.00315/) (2024)
- [Can TabPFN Compete with GNNs for Node Classification via Graph Tabularization?](https://mlsanity.com/paper/2512.08798/) (2025)
- [Learning Posterior Predictive Distributions for Node Classification from Synthetic Graph Priors](https://mlsanity.com/paper/2604.19028/) (2026)
- [Homophily Enhanced Graph Domain Adaptation](https://mlsanity.com/paper/2505.20089/) (2025)
- [Model Generalization on Text Attribute Graphs: Principles with Large Language Models](https://mlsanity.com/paper/2502.11836/) (2025)
- [Towards A Universal Graph Structural Encoder](https://mlsanity.com/paper/2504.10917/) (2025)
- [KG-LLM-Bench: A Scalable Benchmark for Evaluating LLM Reasoning on Textualized Knowledge Graphs](https://mlsanity.com/paper/2504.07087/) (2025)
- [Graph-KV: Breaking Sequence via Injecting Structural Biases into Large Language Models](https://mlsanity.com/paper/2506.07334/) (2025)
- [Semantic Refinement with LLMs for Graph Representations](https://mlsanity.com/paper/2512.21106/) (2025)
- [View Space: Learning Representation across Arbitrary Graphs](https://mlsanity.com/paper/2512.11561/) (2025)
- [Unveiling Mode Connectivity in Graph Neural Networks](https://mlsanity.com/paper/2502.12608/) (2025)
- [Dynamic Bundling with Large Language Models for Zero-Shot Inference on Text-Attributed Graphs](https://mlsanity.com/paper/2505.17599/) (2025)
- [Towards Text-free Graph Foundation Models: Rethinking Multi-Domain Graph Contrastive Learning](https://mlsanity.com/paper/2506.22510/) (2025)
- [Text Meets Topology: Rethinking Out-of-distribution Detection in Text-Rich Networks](https://mlsanity.com/paper/2508.17690/) (2025)
- [Graph Domain Adaptation via Homophily-Agnostic Reconstructing Structure](https://mlsanity.com/paper/2602.07573/) (2026)
- [Modality-free Graph In-context Alignment](https://mlsanity.com/paper/2603.13434/) (2026)
- [GLIP-OOD: Zero-Shot Graph OOD Detection with Graph Foundation Model](https://mlsanity.com/paper/2504.21186/) (2025)
- [Language Models as Messengers: Enhancing Message Passing in Heterophilic Graph Learning](https://mlsanity.com/paper/2505.19762/) (2025)
- [A Scalable Pretraining Framework for Link Prediction with Efficient Adaptation](https://mlsanity.com/paper/2508.04645/) (2025)
- [Graph is a Natural Regularization: Revisiting Vector Quantization for Graph Representation Learning](https://mlsanity.com/paper/2508.06588/) (2025)
- [Node4All: Learning Node Representation Beyond Datasets](https://mlsanity.com/paper/2607.17272/) (2026)
- [SaVe-TAG: LLM-based Interpolation for Long-Tailed Text-Attributed Graphs](https://mlsanity.com/paper/2410.16882/) (2024)
- [Relation-Aware Graph Foundation Model](https://mlsanity.com/paper/2505.12027/) (2025)
- [A Systematic Study of Model Extraction Attacks on Graph Foundation Models](https://mlsanity.com/paper/2511.11912/) (2025)
- [Learning Hierarchical Knowledge in Text-Rich Networks with Taxonomy-Informed Representation Learning](https://mlsanity.com/paper/2603.08159/) (2026)
- [Enhancing LLMs for Graph Tasks via Graph-aware LoRA Generation](https://mlsanity.com/paper/2606.22429/) (2026)

## Abstract (arXiv)

Given the ubiquity of graph data and its applications in diverse domains, building a Graph Foundation Model (GFM) that can work well across different graphs and tasks with a unified backbone has recently garnered significant interests. A major obstacle to achieving this goal stems from the fact that graphs from different domains often exhibit diverse node features. Inspired by multi-modal models that align different modalities with natural language, the text has recently been adopted to provide a unified feature space for diverse graphs. Despite the great potential of these text-space GFMs, current research in this field is hampered by two problems. First, the absence of a comprehensive benchmark with unified problem settings hinders a clear understanding of the comparative effectiveness and practical value of different text-space GFMs. Second, there is a lack of sufficient datasets to thoroughly explore the methods' full potential and verify their effectiveness across diverse settings. To address these issues, we conduct a comprehensive benchmark providing novel text-space datasets and comprehensive evaluation under unified problem settings. Empirical results provide new insights and inspire future research directions. Our code and data are publicly available from urlhttps://github.com/CurryTang/TSGFM.