mlsanityData to 14 Oct 2026

Train Your Own GNN Teacher: Graph-Aware Distillation on Textual Graphs

arXiv 2304.10668 ↗First posted 20 Apr 2023cs.LGcited by 29

Not on the map. Why?

Abstract (arXiv)

How can we learn effective node representations on textual graphs? Graph Neural Networks (GNNs) that use Language Models (LMs) to encode textual information of graphs achieve state-of-the-art performance in many node classification tasks. Yet, combining GNNs with LMs has not been widely explored for practical deployments due to its scalability issues. In this work, we tackle this challenge by developing a Graph-Aware Distillation framework (GRAD) to encode graph structures into an LM for graph-free, fast inference. Different from conventional knowledge distillation, GRAD jointly optimizes a GNN teacher and a graph-free student over the graph's nodes via a shared LM. This encourages the graph-free student to exploit graph information encoded by the GNN teacher while at the same time, enables the GNN teacher to better leverage textual information from unlabeled nodes. As a result, the teacher and the student models learn from each other to improve their overall performance. Experiments in eight node classification benchmarks in both transductive and inductive settings showcase GRAD's superiority over existing distillation approaches for textual graphs.

Key facts. Cited by 29 papers in the map (data through 14 Oct 2026). Builds on 46 of its 54 references that are in the map. First posted on arXiv 20 Apr 2023.

Builds on

46 of 54 references.

References · 46

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  8. 2017Graph Attention Networkscited by 8,923
  9. 2017Inductive Representation Learning on Large Graphscited by 6,320
  10. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  11. 2017Neural Message Passing for Quantum Chemistrycited by 3,487
  12. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
34 more
  1. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,163
  2. 2017Modeling Relational Data with Graph Convolutional Networkscited by 1,694
  3. 2020Knowledge Distillation: A Surveycited by 1,603
  4. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
  5. 2019GraphSAINT: Graph Sampling Based Inductive Learning Methodcited by 618
  6. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 497
  7. 2018Graph Convolutional Networks for Text Classificationcited by 345
  8. 2022LinkBERT: Pretraining Language Models with Document Linkscited by 211
  9. 2020Combining Label Propagation and Simple Models Out-performs Graph Neural Networkscited by 202
  10. 2020Graph-Bert: Only Attention is Needed for Learning Graph Representationscited by 177
  11. 2022GreaseLM: Graph REASoning Enhanced Language Models for Question Answeringcited by 171
  12. 2021Training Graph Neural Networks with 1000 Layerscited by 169
  13. 2022Deep Bidirectional Language-Knowledge Graph Pretrainingcited by 156
  14. 2022Learning on Large-scale Text-attributed Graphs via Variational Inferencecited by 155
  15. 2021Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillationcited by 152
  16. 2021GraphFormers: GNN-nested Transformers for Representation Learning on Textual Graphcited by 143
  17. 2020DistDGL: Distributed Graph Neural Network Training for Billion-Scale Graphscited by 139
  18. 2021Node Feature Extraction by Self-Supervised Multi-scale Neighborhood Predictioncited by 138
  19. 2020Distilling Knowledge from Graph Convolutional Networkscited by 129
  20. 2020A Survey on Negative Transfercited by 122
  21. 2021Graph-MLP: Node Classification without Message Passing in Graphcited by 73
  22. 2021Extract the Knowledge of Graph Neural Networks and Go Beyond it: An Effective Knowledge Distillation Frameworkcited by 69
  23. 2019Text Level Graph Neural Network for Text Classificationcited by 60
  24. 2021Cold Brew: Distilling Graph Node Representations with Incomplete or Missing Neighborhoodscited by 58
  25. 2021TextGNN: Improving Text Encoder via Graph Neural Network in Sponsored Searchcited by 54
  26. 2021Scalable and Adaptive Graph Neural Networks with Self-Label-Enhanced trainingcited by 54
  27. 2021Graph-Free Knowledge Distillation for Graph Neural Networkscited by 46
  28. 2020GraphSAIL: Graph Structure Aware Incremental Learning for Recommender Systemscited by 42
  29. 2021AdsGNN: Behavior-Graph Augmented Relevance Modeling in Sponsored Searchcited by 38
  30. 2020Rethinking Graph Regularization for Graph Neural Networkscited by 36
  31. 2022ReaRev: Adaptive Reasoning for Question Answering over Knowledge Graphscited by 16
  32. 2022Node Representation Learning in Graph via Node-to-Neighbourhood Mutual Information Maximizationcited by 9
  33. 2022Efficient and effective training of language and graph neural network modelscited by 9
  34. 2022E2EG: End-to-End Node Classification Using Graph Topology and Text-based Node Attributescited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 9; 2024: 10; 2025: 8; 2026: 2.

29 papers.

Sort:

  1. 2023Language is All a Graph Needscited by 220
  2. 2023Large Language Models on Graphs: A Comprehensive Surveycited by 192
  3. 2023Graph Foundation Models: Concepts, Opportunities and Challengescited by 103
  4. 2023A Survey of Graph Meets Large Language Model: Progress and Future Directionscited by 85
  5. 2023SimTeG: A Frustratingly Simple Approach Improves Textual Graph Learningcited by 73
  6. 2024Can GNN be Good Adapter for LLMs?cited by 71
  7. 2024Can we Soft Prompt LLMs for Graph Learning Tasks?cited by 52
  8. 2024Graph Machine Learning in the Era of Large Language Models (LLMs)cited by 33
  9. 2023GRENADE: Graph-Centric Language Model for Self-Supervised Representation Learning on Text-Attributed Graphscited by 28
  10. 2025LLM as GNN: Graph Vocabulary Learning for Text-Attributed Graph Foundation Modelscited by 22
  11. 2024Advancing Graph Representation Learning with Large Language Models: A Comprehensive Survey of Techniquescited by 16
  12. 2023When Large Language Models Meet Citation: A Surveycited by 6
17 more
  1. 2025Large Language Models Meet Text-Attributed Graphs: A Survey of Integration Frameworks and Applicationscited by 6
  2. 2023Efficient End-to-end Language Model Fine-tuning on Graphscited by 5
  3. 2024Parameter-Efficient Tuning Large Language Models for Graph Representation Learningcited by 3
  4. 2024Verbalized Graph Representation Learning: A Fully Interpretable Graph Model Based on Large Language Models Throughout the Entire Processcited by 3
  5. 2024The Graph's Apprentice: Teaching an LLM Low Level Knowledge for Circuit Quality Estimationcited by 3
  6. 2024Large Language Models Meet Graph Neural Networks: A Perspective of Graph Miningcited by 3
  7. 2025Integrating Structural and Semantic Signals in Text-Attributed Graphs with BiGTexcited by 3
  8. 2025Adversarial Attacks and Defenses on Graph-aware Large Language Models (LLMs)cited by 3
  9. 2025Language Models as Messengers: Enhancing Message Passing in Heterophilic Graph Learningcited by 2
  10. 2024SemPool: Simple, robust, and interpretable KG pooling for enhancing language modelscited by 1
  11. 2024Towards Versatile Graph Learning Approach: from the Perspective of Large Language Modelscited by 1
  12. 2025From Text to Graph: Leveraging Graph Neural Networks for Enhanced Explainability in NLPcited by 1
  13. 2023Graph Coordinates and Conventional Neural Networks -- An Alternative for Graph Neural Networkscited by 0
  14. 2025Fusing Knowledge and Language: A Comparative Study of Knowledge Graph-Based Question Answering with LLMscited by 0
  15. 2025GraphMatch: Fusing Language and Graph Representations in a Dynamic Two-Sided Work Marketplacecited by 0
  16. 2026Interpretable Unsupervised Community Detection with LLM-Symbolized Structured Processescited by 0
  17. 2026TemporalGraphLLM: Temporal Graph Neural Networks with Large Language Models for Dynamic Text-Attributed Graphscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Enhancing Graph Neural Networks: A Mutual Learning Approach
  2. 2023VQGraph: Rethinking Graph Representation Space for Bridging GNNs and MLPs
  3. 2023Unveiling the Unseen Potential of Graph Learning through MLPs: Effective Graph Learners Using Propagation-Embracing MLPs
  4. 2024A Teacher-Free Graph Knowledge Distillation Framework with Dual Self-Distillation
  5. 2025Preference-driven Knowledge Distillation for Few-shot Node Classification
  6. 2023Propagate & Distill: Towards Effective Graph Learners Using Propagation-Embracing MLPs
  7. 2021On Representation Knowledge Distillation for Graph Neural Networks
  8. 2022Data-Free Adversarial Knowledge Distillation for Graph Neural Networks