# GOFA: A Generative One-For-All Model for Joint Graph Language Modeling

arXiv: https://arxiv.org/abs/2407.09709 · HTML: https://mlsanity.com/paper/2407.09709/

## Key facts

- Cited by 57 papers in the map (data through 14 Oct 2026).
- Builds on 60 of its 76 references that are in the map.
- First posted on arXiv 12 Jul 2024.

## Builds on (most cited first, 30 of 60)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Fast Graph Representation Learning with PyTorch Geometric](https://mlsanity.com/paper/1903.02428/) (2019)
- [Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understanding](https://mlsanity.com/paper/2306.02858/) (2023)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [Deep Graph Infomax](https://mlsanity.com/paper/1809.10341/) (2018)
- [Weisfeiler and Leman Go Neural: Higher-order Graph Neural Networks](https://mlsanity.com/paper/1810.02244/) (2018)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [Galactica: A Large Language Model for Science](https://mlsanity.com/paper/2211.09085/) (2022)
- [Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classification](https://mlsanity.com/paper/2009.03509/) (2020)
- [OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphs](https://mlsanity.com/paper/2103.09430/) (2021)
- [Graph Self-Supervised Learning: A Survey](https://mlsanity.com/paper/2103.00111/) (2021)
- [Wiki-CS: A Wikipedia-Based Benchmark for Graph Neural Networks](https://mlsanity.com/paper/2007.02901/) (2020)
- [One for All: Towards Training One Graph Model for All Classification Tasks](https://mlsanity.com/paper/2310.00149/) (2023)
- [G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering](https://mlsanity.com/paper/2402.07630/) (2024)
- [Language is All a Graph Needs](https://mlsanity.com/paper/2308.07134/) (2023)
- [Sequential Modeling Enables Scalable Learning for Large Vision Models](https://mlsanity.com/paper/2312.00785/) (2023)
- [In-context Autoencoder for Context Compression in a Large Language Model](https://mlsanity.com/paper/2307.06945/) (2023)
- [Large Language Models on Graphs: A Comprehensive Survey](https://mlsanity.com/paper/2312.02783/) (2023)
- [Universal Invariant and Equivariant Graph Neural Networks](https://mlsanity.com/paper/1905.04943/) (2019)
- [LLaGA: Large Language and Graph Assistant](https://mlsanity.com/paper/2402.08170/) (2024)

## Built on by (most cited first, 30 of 57)

- [Retrieval-Augmented Generation with Graphs (GraphRAG)](https://mlsanity.com/paper/2501.00309/) (2024)
- [When Do LLMs Help With Node Classification? A Comprehensive Analysis](https://mlsanity.com/paper/2502.00829/) (2025)
- [Graph is a Substrate Across Data Modalities](https://mlsanity.com/paper/2601.22384/) (2026)
- [One Prompt Fits All: Universal Graph Adaptation for Pretrained Models](https://mlsanity.com/paper/2509.22416/) (2025)
- [Graph Learning in the Era of LLMs: A Survey from the Perspective of Data, Models, and Tasks](https://mlsanity.com/paper/2412.12456/) (2024)
- [Learning Posterior Predictive Distributions for Node Classification from Synthetic Graph Priors](https://mlsanity.com/paper/2604.19028/) (2026)
- [Model Generalization on Text Attribute Graphs: Principles with Large Language Models](https://mlsanity.com/paper/2502.11836/) (2025)
- [Attention Mechanisms Perspective: Exploring LLM Processing of Graph-Structured Data](https://mlsanity.com/paper/2505.02130/) (2025)
- [Enhance Graph Alignment for Large Language Models](https://mlsanity.com/paper/2410.11370/) (2024)
- [Towards A Universal Graph Structural Encoder](https://mlsanity.com/paper/2504.10917/) (2025)
- [UniGTE: Unified Graph-Text Encoding for Zero-Shot Generalization across Graph Tasks and Domains](https://mlsanity.com/paper/2510.16885/) (2025)
- [Toward Effective Multimodal Graph Foundation Model: A Divide-and-Conquer Based Approach](https://mlsanity.com/paper/2602.04116/) (2026)
- [Toward Graph-Tokenizing Large Language Models with Reconstructive Graph Instruction Tuning](https://mlsanity.com/paper/2603.01385/) (2026)
- [Graph-KV: Breaking Sequence via Injecting Structural Biases into Large Language Models](https://mlsanity.com/paper/2506.07334/) (2025)
- [Semantic Refinement with LLMs for Graph Representations](https://mlsanity.com/paper/2512.21106/) (2025)
- [Dynamic Bundling with Large Language Models for Zero-Shot Inference on Text-Attributed Graphs](https://mlsanity.com/paper/2505.17599/) (2025)
- [Modality-free Graph In-context Alignment](https://mlsanity.com/paper/2603.13434/) (2026)
- [Learning Graph Foundation Models on Riemannian Graph-of-Graphs](https://mlsanity.com/paper/2605.09993/) (2026)
- [Towards Data-centric Machine Learning on Directed Graphs: a Survey](https://mlsanity.com/paper/2412.01849/) (2024)
- [Bootstrapping Heterogeneous Graph Representation Learning via Large Language Models: A Generalized Approach](https://mlsanity.com/paper/2412.08038/) (2024)
- [A Survey of Cross-domain Graph Learning: Progress and Future Directions](https://mlsanity.com/paper/2503.11086/) (2025)
- [GILT: An LLM-Free, Tuning-Free Graph Foundational Model for In-Context Learning](https://mlsanity.com/paper/2510.04567/) (2025)
- [See or Say Graphs: Agent-Driven Scalable Graph Structure Understanding with Vision-Language Models](https://mlsanity.com/paper/2510.16769/) (2025)
- [Rethinking Federated Graph Foundation Models: A Graph-Language Alignment-based Approach](https://mlsanity.com/paper/2601.21369/) (2026)
- [Revisiting Graph-Tokenizing Large Language Models: A Systematic Evaluation of Graph Token Understanding](https://mlsanity.com/paper/2605.03514/) (2026)
- [Can Large Language Models Act as Ensembler for Multi-GNNs?](https://mlsanity.com/paper/2410.16822/) (2024)
- [Relation-Aware Graph Foundation Model](https://mlsanity.com/paper/2505.12027/) (2025)
- [When LLMs meet open-world graph learning: a new perspective for unlabeled data uncertainty](https://mlsanity.com/paper/2505.13989/) (2025)
- [Beyond One-Size-Fits-All: Adaptive Subgraph Denoising for Zero-Shot Graph Learning with Large Language Models](https://mlsanity.com/paper/2603.02938/) (2026)
- [LLM as Graph Kernel: Rethinking Message Passing on Text-Rich Graphs](https://mlsanity.com/paper/2603.14937/) (2026)

## Abstract (arXiv)

Foundation models, such as Large Language Models (LLMs) or Large Vision Models (LVMs), have emerged as one of the most powerful tools in the respective fields. However, unlike text and image data, graph data do not have a definitive structure, posing great challenges to developing a Graph Foundation Model (GFM). For example, current attempts at designing general graph models either transform graph data into a language format for LLM-based prediction or still train a GNN model with LLM as an assistant. The former can handle unlimited tasks, while the latter captures graph structure much better -- yet, no existing work can achieve both simultaneously. In this paper, we identify three key desirable properties of a GFM: self-supervised pretraining, fluidity in tasks, and graph awareness. To account for these properties, we extend the conventional language modeling to the graph domain and propose a novel generative graph language model GOFA to solve the problem. The model interleaves randomly initialized GNN layers into a frozen pre-trained LLM so that the semantic and structural modeling abilities are organically combined. GOFA is pre-trained on newly proposed graph-level next-word prediction, question-answering, and structural tasks to obtain the above GFM properties. The pre-trained model is further fine-tuned on downstream tasks to obtain task-solving ability. The fine-tuned model is evaluated on various downstream tasks, demonstrating a strong ability to solve structural and contextual problems in zero-shot scenarios. The code is available at https://github.com/JiaruiFeng/GOFA.