# Graph-oriented Instruction Tuning of Large Language Models for Generic Graph Mining

arXiv: https://arxiv.org/abs/2403.04780 · HTML: https://mlsanity.com/paper/2403.04780/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 77 references that are in the map.
- First posted on arXiv 2 Mar 2024.

## Builds on (most cited first, 30 of 48)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Open Graph Benchmark: Datasets for Machine Learning on Graphs](https://mlsanity.com/paper/2005.00687/) (2020)
- [node2vec: Scalable Feature Learning for Networks](https://mlsanity.com/paper/1607.00653/) (2016)
- [Modeling Relational Data with Graph Convolutional Networks](https://mlsanity.com/paper/1703.06103/) (2017)
- [Baichuan 2: Open Large-scale Language Models](https://mlsanity.com/paper/2309.10305/) (2023)
- [Heterogeneous Graph Transformer](https://mlsanity.com/paper/2003.01332/) (2020)
- [How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources](https://mlsanity.com/paper/2306.04751/) (2023)
- [A Comprehensive Survey of Graph Embedding: Problems, Techniques and Applications](https://mlsanity.com/paper/1709.07604/) (2017)
- [On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselines](https://mlsanity.com/paper/2006.04884/) (2020)
- [Can Language Models Solve Graph Problems in Natural Language?](https://mlsanity.com/paper/2305.10037/) (2023)
- [One for All: Towards Training One Graph Model for All Classification Tasks](https://mlsanity.com/paper/2310.00149/) (2023)
- [GraphGPT: Graph Instruction Tuning for Large Language Models](https://mlsanity.com/paper/2310.13023/) (2023)
- [Language is All a Graph Needs](https://mlsanity.com/paper/2308.07134/) (2023)
- [GPT4Graph: Can Large Language Models Understand Graph Structured Data ? An Empirical Evaluation and Benchmarking](https://mlsanity.com/paper/2305.15066/) (2023)
- [Are we really making much progress? Revisiting, benchmarking, and refining heterogeneous graph neural networks](https://mlsanity.com/paper/2112.14936/) (2021)
- [Training Graph Neural Networks with 1000 Layers](https://mlsanity.com/paper/2106.07476/) (2021)
- [Distilling Reasoning Capabilities into Smaller Language Models](https://mlsanity.com/paper/2212.00193/) (2022)
- [All in One: Multi-task Prompting for Graph Neural Networks](https://mlsanity.com/paper/2307.01504/) (2023)
- [Harnessing Explanations: LLM-to-LM Interpreter for Enhanced Text-Attributed Graph Representation Learning](https://mlsanity.com/paper/2305.19523/) (2023)

## Built on by (most cited first, 20 of 20)

- [A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Models](https://mlsanity.com/paper/2501.13958/) (2025)
- [Plan-on-Graph: Self-Correcting Adaptive Planning of Large Language Model on Knowledge Graphs](https://mlsanity.com/paper/2410.23875/) (2024)
- [A Survey of Large Language Models for Graphs](https://mlsanity.com/paper/2405.08011/) (2024)
- [Knowledge Graph Large Language Model (KG-LLM) for Link Prediction](https://mlsanity.com/paper/2403.07311/) (2024)
- [How Do Large Language Models Understand Graph Patterns? A Benchmark for Graph Pattern Comprehension](https://mlsanity.com/paper/2410.05298/) (2024)
- [Graph-based Approaches and Functionalities in Retrieval-Augmented Generation: A Comprehensive Survey](https://mlsanity.com/paper/2504.10499/) (2025)
- [Graph Learning in the Era of LLMs: A Survey from the Perspective of Data, Models, and Tasks](https://mlsanity.com/paper/2412.12456/) (2024)
- [Towards Incremental Learning in Large Language Models: A Critical Review](https://mlsanity.com/paper/2404.18311/) (2024)
- [Large Language Models Meet Text-Attributed Graphs: A Survey of Integration Frameworks and Applications](https://mlsanity.com/paper/2510.21131/) (2025)
- [How to Make LMs Strong Node Classifiers?](https://mlsanity.com/paper/2410.02296/) (2024)
- [Integrating Structural and Semantic Signals in Text-Attributed Graphs with BiGTex](https://mlsanity.com/paper/2504.12474/) (2025)
- [Bootstrapping Heterogeneous Graph Representation Learning via Large Language Models: A Generalized Approach](https://mlsanity.com/paper/2412.08038/) (2024)
- [Towards Open-World Retrieval-Augmented Generation on Knowledge Graph: A Multi-Agent Collaboration Framework](https://mlsanity.com/paper/2509.01238/) (2025)
- [Revisiting Graph-Tokenizing Large Language Models: A Systematic Evaluation of Graph Token Understanding](https://mlsanity.com/paper/2605.03514/) (2026)
- [Relation-Aware Graph Foundation Model](https://mlsanity.com/paper/2505.12027/) (2025)
- [Colorful Talks with Graphs: Human-Interpretable Graph Encodings for Large Language Models](https://mlsanity.com/paper/2602.10386/) (2026)
- [Bridging the Divide: End-to-End Sequence-Graph Learning](https://mlsanity.com/paper/2510.25126/) (2025)
- [Semi-supervised Instruction Tuning for Large Language Models on Text-Attributed Graphs](https://mlsanity.com/paper/2601.12807/) (2026)
- [NAG: A Unified Native Architecture for Encoder-free Text-Graph Modeling in Language Models](https://mlsanity.com/paper/2601.22657/) (2026)
- [UNIT: Unleash Large Language Models Potential for Graph Continual Learning](https://mlsanity.com/paper/2607.10159/) (2026)

## Abstract (arXiv)

Graphs with abundant attributes are essential in modeling interconnected entities and enhancing predictions across various real-world applications. Traditional Graph Neural Networks (GNNs) often require re-training for different graph tasks and datasets. Although the emergence of Large Language Models (LLMs) has introduced new paradigms in natural language processing, their potential for generic graph mining, training a single model to simultaneously handle diverse tasks and datasets, remains under-explored. To this end, our novel framework MuseGraph, seamlessly integrates the strengths of GNNs and LLMs into one foundation model for graph mining across tasks and datasets. This framework first features a compact graph description to encapsulate key graph information within language token limitations. Then, we propose a diverse instruction generation mechanism with Chain-of-Thought (CoT)-based instruction packages to distill the reasoning capabilities from advanced LLMs like GPT-4. Finally, we design a graph-aware instruction tuning strategy to facilitate mutual enhancement across multiple tasks and datasets while preventing catastrophic forgetting of LLMs' generative abilities. Our experimental results demonstrate significant improvements in five graph tasks and ten datasets, showcasing the potential of our MuseGraph in enhancing the accuracy of graph-oriented downstream tasks while improving the generation abilities of LLMs.