mlsanityData to 14 Oct 2026

Distilling Large Language Models for Text-Attributed Graph Learning

arXiv 2402.12022 ↗First posted 19 Feb 2024cs.CL cs.LGcited by 20

On the map: Knowledge distillation → Graph neural network distillation

Abstract (arXiv)

Text-Attributed Graphs (TAGs) are graphs of connected textual documents. Graph models can efficiently learn TAGs, but their training heavily relies on human-annotated labels, which are scarce or even unavailable in many applications. Large language models (LLMs) have recently demonstrated remarkable capabilities in few-shot and zero-shot TAG learning, but they suffer from scalability, cost, and privacy issues. Therefore, in this work, we focus on synergizing LLMs and graph models with their complementary strengths by distilling the power of LLMs to a local graph model on TAG learning. To address the inherent gaps between LLMs (generative models for texts) and graph models (discriminative models for graphs), we propose first to let LLMs teach an interpreter with rich textual rationale and then let a student model mimic the interpreter's reasoning without LLMs' textual rationale. Extensive experiments validate the efficacy of our proposed framework.

Key facts. Cited by 20 papers in the map (data through 14 Oct 2026). Placed in Graph neural network distillation, one of 12 directions in Knowledge distillation. Builds on 31 of its 53 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

31 of 53 references.

References · 31

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  3. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  4. 2017Graph Attention Networkscited by 8,923
  5. 2017Inductive Representation Learning on Large Graphscited by 6,320
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  7. 2019Fast Graph Representation Learning with PyTorch Geometriccited by 2,434
  8. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  9. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,163
  10. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,167
  11. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
  12. 2022Large Language Models Are Reasoning Teacherscited by 329
19 more
  1. 2022Teaching Small Language Models to Reasoncited by 284
  2. 2023Exploring the Potential of Large Language Models (LLMs) in Learning on Graphscited by 274
  3. 2015Unifying distillation and privileged informationcited by 233
  4. 2023Language is All a Graph Needscited by 220
  5. 2024LLaGA: Large Language and Graph Assistantcited by 181
  6. 2022Distilling Reasoning Capabilities into Smaller Language Modelscited by 158
  7. 2022Learning on Large-scale Text-attributed Graphs via Variational Inferencecited by 155
  8. 2021Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillationcited by 152
  9. 2023Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Stepcited by 148
  10. 2021Node Feature Extraction by Self-Supervised Multi-scale Neighborhood Predictioncited by 138
  11. 2022Explanations from Large Language Models Make Small Reasoners Bettercited by 124
  12. 2023SimTeG: A Frustratingly Simple Approach Improves Textual Graph Learningcited by 73
  13. 2023Label-free Node Classification on Graphs with Large Language Models (LLMS)cited by 72
  14. 2021TextGNN: Improving Text Encoder via Graph Neural Network in Sponsored Searchcited by 54
  15. 2023Large Language Models as Topological Structure Enhancers for Text-Attributed Graphscited by 27
  16. 2023Beyond Text: A Deep Dive into Large Language Models' Ability on Understanding Graph Datacited by 25
  17. 2021Zero-shot Node Classification with Decomposed Graph Prototype Networkcited by 15
  18. 2023XAI-CLASS: Explanation-Enhanced Text Classification with Extremely Weak Supervisioncited by 3
  19. 2023Visual Attention Prompted Prediction and Learningcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 10; 2026: 3.

20 papers.

Sort:

By area:

  1. 2024GRAG: Graph Retrieval-Augmented Generationcited by 78
  2. 2024DTGB: A Comprehensive Benchmark for Dynamic Text-Attributed Graphscited by 27
  3. 2025Efficient Multivariate Time Series Forecasting via Calibrated Language Models with Privileged Knowledge Distillationcited by 20
  4. 2024ELAD: Explanation-Guided Large Language Models Active Distillationcited by 12
  5. 2025Graph2text or Graph2token: A Perspective of Large Language Models for Graph Learningcited by 12
  6. 2024Exploring the Potential of Large Language Models for Heterophilic Graphscited by 10
  7. 2025Can LLMs Find Fraudsters? Multi-level LLM Enhanced Graph Fraud Detectioncited by 10
  8. 2024A Survey on Self-Supervised Graph Foundation Models: Knowledge-Based Perspectivecited by 9
  9. 2025A Survey on Efficient Large Language Model Training: From Data-centric Perspectivescited by 9
  10. 2025LLM-driven Knowledge Distillation for Dynamic Text-Attributed Graphscited by 8
  11. 2025Model Generalization on Text Attribute Graphs: Principles with Large Language Modelscited by 8
  12. 2024GraphNarrator: Generating Textual Explanations for Graph Neural Networkscited by 7
8 more
  1. 2025Toward General and Robust LLM-enhanced Text-attributed Graph Learningcited by 6
  2. 2024How to Make LMs Strong Node Classifiers?cited by 4
  3. 2025Trustworthy GNNs with LLMs: A Systematic Review and Taxonomycited by 1
  4. 2025Preference-driven Knowledge Distillation for Few-shot Node Classificationcited by 1
  5. 2026Integrating Graphs, Large Language Models, and Agents: Reasoning and Retrievalcited by 1
  6. 2026Adaptive Recurrent Message Passing for Test Time Computing on Graphscited by 1
  7. 2025Efficient Text-Attributed Graph Learning through Selective Annotation and Graph Alignmentcited by 0
  8. 2026Detecting Miscitation on the Scholarly Web through LLM-Augmented Text-Rich Graph Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Large Language Models Meet Text-Attributed Graphs: A Survey of Integration Frameworks and Applications
  2. 2026TAG-DLM: Diffusion Language Models for Text-Attributed Graph Learning
  3. 2023Large Language Models on Graphs: A Comprehensive Survey
  4. 2025Exploring Graph Learning Tasks with Pure LLMs: A Comprehensive Benchmark and Investigation
  5. 2024GUNDAM: Aligning Large Language Models with Graph Understanding
  6. 2025Quantizing Text-attributed Graphs for Semantic-Structural Integration
  7. 2025SSTAG: Structure-Aware Self-Supervised Learning Method for Text-Attributed Graphs
  8. 2026Teaching LLMs to See Graphs: Unifying Text and Structural Reasoning