# HiGPT: Heterogeneous Graph Language Model

arXiv: https://arxiv.org/abs/2402.16024 · HTML: https://mlsanity.com/paper/2402.16024/

## Key facts

- Cited by 45 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 45 references that are in the map.
- First posted on arXiv 25 Feb 2024.

## Builds on (most cited first, 30 of 35)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [Heterogeneous Graph Attention Network](https://mlsanity.com/paper/1903.07293/) (2019)
- [Heterogeneous Graph Transformer](https://mlsanity.com/paper/2003.01332/) (2020)
- [MAGNN: Metapath Aggregated Graph Neural Network for Heterogeneous Graph Embedding](https://mlsanity.com/paper/2002.01680/) (2020)
- [Exploring the Potential of Large Language Models (LLMs) in Learning on Graphs](https://mlsanity.com/paper/2307.03393/) (2023)
- [Language is All a Graph Needs](https://mlsanity.com/paper/2308.07134/) (2023)
- [Generated Knowledge Prompting for Commonsense Reasoning](https://mlsanity.com/paper/2110.08387/) (2021)
- [Are we really making much progress? Revisiting, benchmarking, and refining heterogeneous graph neural networks](https://mlsanity.com/paper/2112.14936/) (2021)
- [Talk like a Graph: Encoding Graphs for Large Language Models](https://mlsanity.com/paper/2310.04560/) (2023)
- [LLMRec: Large Language Models with Graph Augmentation for Recommendation](https://mlsanity.com/paper/2311.00423/) (2023)
- [Distilling Reasoning Capabilities into Smaller Language Models](https://mlsanity.com/paper/2212.00193/) (2022)
- [Representation Learning with Large Language Models for Recommendation](https://mlsanity.com/paper/2310.15950/) (2023)
- [Self-supervised Heterogeneous Graph Neural Network with Co-contrastive Learning](https://mlsanity.com/paper/2105.09111/) (2021)
- [A Survey on Heterogeneous Graph Embedding: Methods, Techniques, Applications and Sources](https://mlsanity.com/paper/2011.14867/) (2020)
- [GraphLLM: Boosting Graph Reasoning Ability of Large Language Model](https://mlsanity.com/paper/2310.05845/) (2023)
- [Unsupervised Attributed Multiplex Network Embedding](https://mlsanity.com/paper/1911.06750/) (2019)
- [Augmenting Low-Resource Text Classification with Graph-Grounded Pre-training and Prompting](https://mlsanity.com/paper/2305.03324/) (2023)
- [Relational Message Passing for Knowledge Graph Completion](https://mlsanity.com/paper/2002.06757/) (2020)
- [Heterogeneous Graph Masked Autoencoders](https://mlsanity.com/paper/2208.09957/) (2022)
- [Self-supervised Auxiliary Learning with Meta-paths for Heterogeneous Graphs](https://mlsanity.com/paper/2007.08294/) (2020)
- [Contrastive Meta Learning with Behavior Multiplicity for Recommendation](https://mlsanity.com/paper/2202.08523/) (2022)
- [Query-Dependent Prompt Evaluation and Optimization with Offline Inverse RL](https://mlsanity.com/paper/2309.06553/) (2023)
- [Self-supervised Heterogeneous Graph Pre-training Based on Structural Clustering](https://mlsanity.com/paper/2210.10462/) (2022)
- [Heterogeneous Graph Contrastive Learning for Recommendation](https://mlsanity.com/paper/2303.00995/) (2023)

## Built on by (most cited first, 30 of 45)

- [SAMGPT: Text-free Graph Foundation Model for Multi-domain Pre-training and Cross-domain Adaptation](https://mlsanity.com/paper/2502.05424/) (2025)
- [GOFA: A Generative One-For-All Model for Joint Graph Language Modeling](https://mlsanity.com/paper/2407.09709/) (2024)
- [AnyGraph: Graph Foundation Model in the Wild](https://mlsanity.com/paper/2408.10700/) (2024)
- [RelGNN: Composite Message Passing for Relational Deep Learning](https://mlsanity.com/paper/2502.06784/) (2025)
- [Deep Graph Anomaly Detection: A Survey and New Perspectives](https://mlsanity.com/paper/2409.09957/) (2024)
- [GFM-RAG: Graph Foundation Model for Retrieval Augmented Generation](https://mlsanity.com/paper/2502.01113/) (2025)
- [Graph-oriented Instruction Tuning of Large Language Models for Generic Graph Mining](https://mlsanity.com/paper/2403.04780/) (2024)
- [RecDiff: Diffusion Model for Social Recommendation](https://mlsanity.com/paper/2406.01629/) (2024)
- [GRAVER: Generative Graph Vocabularies for Robust Graph Foundation Models Fine-tuning](https://mlsanity.com/paper/2511.05592/) (2025)
- [One Prompt Fits All: Universal Graph Adaptation for Pretrained Models](https://mlsanity.com/paper/2509.22416/) (2025)
- [Node Identifiers: Compact, Discrete Representations for Efficient Graph Learning](https://mlsanity.com/paper/2405.16435/) (2024)
- [A Survey on Self-Supervised Graph Foundation Models: Knowledge-Based Perspective](https://mlsanity.com/paper/2403.16137/) (2024)
- [Urban Region Pre-training and Prompting: A Graph-based Approach](https://mlsanity.com/paper/2408.05920/) (2024)
- [GraphAgent: Agentic Graph Language Assistant](https://mlsanity.com/paper/2412.17029/) (2024)
- [A Comprehensive Data-centric Overview of Federated Graph Learning](https://mlsanity.com/paper/2507.16541/) (2025)
- [A Comprehensive Survey on Self-Supervised Learning for Recommendation](https://mlsanity.com/paper/2404.03354/) (2024)
- [CodeGraph: Enhancing Graph Reasoning of LLMs with Code](https://mlsanity.com/paper/2408.13863/) (2024)
- [DiffGraph: Heterogeneous Graph Diffusion Model](https://mlsanity.com/paper/2501.02313/) (2025)
- [Bridging Visualization and Optimization: Multimodal Large Language Models on Graph-Structured Combinatorial Optimization](https://mlsanity.com/paper/2501.11968/) (2025)
- [Multi-Scale Heterogeneous Text-Attributed Graph Datasets From Diverse Domains](https://mlsanity.com/paper/2412.08937/) (2024)
- [GraphChain: Large Language Models for Large-scale Graph Analysis via Tool Chaining](https://mlsanity.com/paper/2511.00457/) (2025)
- [From Anchors to Answers: A Novel Node Tokenizer for Integrating Graph Structure into Large Language Models](https://mlsanity.com/paper/2410.10743/) (2024)
- [RAG-GFM: Overcoming In-Memory Bottlenecks in Graph Foundation Models via Retrieval-Augmented Generation](https://mlsanity.com/paper/2601.15124/) (2026)
- [Large Language Models Meet Graph Neural Networks: A Perspective of Graph Mining](https://mlsanity.com/paper/2412.19211/) (2024)
- [Towards Text-free Graph Foundation Models: Rethinking Multi-Domain Graph Contrastive Learning](https://mlsanity.com/paper/2506.22510/) (2025)
- [STAGE: Tackling Semantic Drift in Multimodal Federated Graph Learning](https://mlsanity.com/paper/2605.11919/) (2026)
- [A Survey of Cross-domain Graph Learning: Progress and Future Directions](https://mlsanity.com/paper/2503.11086/) (2025)
- [Towards Efficient LLM-aware Heterogeneous Graph Learning](https://mlsanity.com/paper/2511.17923/) (2025)
- [HeTa: Relation-wise Heterogeneous Graph Foundation Attack Model](https://mlsanity.com/paper/2506.07428/) (2025)
- [Node Role-Guided LLMs for Dynamic Graph Clustering](https://mlsanity.com/paper/2603.13799/) (2026)

## Abstract (arXiv)

Heterogeneous graph learning aims to capture complex relationships and diverse relational semantics among entities in a heterogeneous graph to obtain meaningful representations for nodes and edges. Recent advancements in heterogeneous graph neural networks (HGNNs) have achieved state-of-the-art performance by considering relation heterogeneity and using specialized message functions and aggregation rules. However, existing frameworks for heterogeneous graph learning have limitations in generalizing across diverse heterogeneous graph datasets. Most of these frameworks follow the "pre-train" and "fine-tune" paradigm on the same dataset, which restricts their capacity to adapt to new and unseen data. This raises the question: "Can we generalize heterogeneous graph models to be well-adapted to diverse downstream learning tasks with distribution shifts in both node token sets and relation type heterogeneity?'' To tackle those challenges, we propose HiGPT, a general large graph model with Heterogeneous graph instruction-tuning paradigm. Our framework enables learning from arbitrary heterogeneous graphs without the need for any fine-tuning process from downstream datasets. To handle distribution shifts in heterogeneity, we introduce an in-context heterogeneous graph tokenizer that captures semantic relationships in different heterogeneous graphs, facilitating model adaptation. We incorporate a large corpus of heterogeneity-aware graph instructions into our HiGPT, enabling the model to effectively comprehend complex relation heterogeneity and distinguish between various types of graph tokens. Furthermore, we introduce the Mixture-of-Thought (MoT) instruction augmentation paradigm to mitigate data scarcity by generating diverse and informative instructions. Through comprehensive evaluations, our proposed framework demonstrates exceptional performance in terms of generalization performance.