# Boosting Graph Neural Networks via Adaptive Knowledge Distillation

arXiv: https://arxiv.org/abs/2210.05920 · HTML: https://mlsanity.com/paper/2210.05920/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Graph neural network distillation, one of 12 directions in Knowledge distillation.
- Builds on 42 of its 61 references that are in the map.
- First posted on arXiv 12 Oct 2022.

## Placed in

- Knowledge distillation / [Graph neural network distillation](https://mlsanity.com/distill/graph-neural-network-distillation/)

## Builds on (most cited first, 30 of 42)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [Fast Graph Representation Learning with PyTorch Geometric](https://mlsanity.com/paper/1903.02428/) (2019)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Simplifying Graph Convolutional Networks](https://mlsanity.com/paper/1902.07153/) (2019)
- [Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning](https://mlsanity.com/paper/1801.07606/) (2018)
- [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)](https://mlsanity.com/paper/1511.07289/) (2015)
- [Pitfalls of Graph Neural Network Evaluation](https://mlsanity.com/paper/1811.05868/) (2018)
- [Predict then Propagate: Graph Neural Networks meet Personalized PageRank](https://mlsanity.com/paper/1810.05997/) (2018)
- [Strategies for Pre-training Graph Neural Networks](https://mlsanity.com/paper/1905.12265/) (2019)
- [Contrastive Representation Distillation](https://mlsanity.com/paper/1910.10699/) (2019)
- [Image-based Recommendations on Styles and Substitutes](https://mlsanity.com/paper/1506.04757/) (2015)
- [On the Bottleneck of Graph Neural Networks and its Practical Implications](https://mlsanity.com/paper/2006.05205/) (2020)
- [Born Again Neural Networks](https://mlsanity.com/paper/1805.04770/) (2018)
- [A Comprehensive Overhaul of Feature Distillation](https://mlsanity.com/paper/1904.01866/) (2019)
- [PairNorm: Tackling Oversmoothing in GNNs](https://mlsanity.com/paper/1909.12223/) (2019)
- [Do Wide and Deep Networks Learn the Same Things? Uncovering How Neural Network Representations Vary with Width and Depth](https://mlsanity.com/paper/2010.15327/) (2020)
- [Does Knowledge Distillation Really Work?](https://mlsanity.com/paper/2106.05945/) (2021)
- [Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillation](https://mlsanity.com/paper/2110.08727/) (2021)
- [Distilling Knowledge from Graph Convolutional Networks](https://mlsanity.com/paper/2003.10477/) (2020)
- [Few-Shot Graph Learning for Molecular Property Prediction](https://mlsanity.com/paper/2102.07916/) (2021)
- [Adaptive Multi-Teacher Multi-level Knowledge Distillation](https://mlsanity.com/paper/2103.04062/) (2021)

## Built on by (most cited first, 14 of 14)

- [Beyond Answers: Transferring Reasoning Capabilities to Smaller LLMs Using Multi-Teacher Knowledge Distillation](https://mlsanity.com/paper/2402.04616/) (2024)
- [Knowledge Distillation on Graphs: A Survey](https://mlsanity.com/paper/2302.00219/) (2023)
- [Data-centric Graph Learning: A Survey](https://mlsanity.com/paper/2310.04987/) (2023)
- [LIX: Implicitly Infusing Spatial Geometric Prior Knowledge into Visual Semantic Segmentation for Autonomous Driving](https://mlsanity.com/paper/2403.08215/) (2024)
- [COMBHelper: A Neural Approach to Reduce Search Space for Graph Combinatorial Problems](https://mlsanity.com/paper/2312.09086/) (2023)
- [Classifying Nodes in Graphs without GNNs](https://mlsanity.com/paper/2402.05934/) (2024)
- [Shared Growth of Graph Neural Networks via Prompted Free-direction Knowledge Distillation](https://mlsanity.com/paper/2307.00534/) (2023)
- [Low-Dimensional Federated Knowledge Graph Embedding via Knowledge Distillation](https://mlsanity.com/paper/2408.05748/) (2024)
- [Toward Fair Graph Neural Networks Via Dual-Teacher Knowledge Distillation](https://mlsanity.com/paper/2412.00382/) (2024)
- [Scaled Supervision is an Implicit Lipschitz Regularizer](https://mlsanity.com/paper/2503.14813/) (2025)
- [Preference-driven Knowledge Distillation for Few-shot Node Classification](https://mlsanity.com/paper/2510.10116/) (2025)
- [Pruned Traffic Trees: Native Semantic Compression with a Protocol-Structured Model Family for Encrypted Traffic Classification](https://mlsanity.com/paper/2608.21874/) (2026)
- [Enhancing Transformer with GNN Structural Knowledge via Distillation: A Novel Approach](https://mlsanity.com/paper/2503.01888/) (2025)
- [Enhancing Graph Neural Networks: A Mutual Learning Approach](https://mlsanity.com/paper/2510.19223/) (2025)

## Abstract (arXiv)

Graph neural networks (GNNs) have shown remarkable performance on diverse graph mining tasks. Although different GNNs can be unified as the same message passing framework, they learn complementary knowledge from the same graph. Knowledge distillation (KD) is developed to combine the diverse knowledge from multiple models. It transfers knowledge from high-capacity teachers to a lightweight student. However, to avoid oversmoothing, GNNs are often shallow, which deviates from the setting of KD. In this context, we revisit KD by separating its benefits from model compression and emphasizing its power of transferring knowledge. To this end, we need to tackle two challenges: how to transfer knowledge from compact teachers to a student with the same capacity; and, how to exploit student GNN's own strength to learn knowledge. In this paper, we propose a novel adaptive KD framework, called BGNN, which sequentially transfers knowledge from multiple GNNs into a student GNN. We also introduce an adaptive temperature module and a weight boosting module. These modules guide the student to the appropriate knowledge for effective learning. Extensive experiments have demonstrated the effectiveness of BGNN. In particular, we achieve up to 3.05% improvement for node classification and 6.35% improvement for graph classification over vanilla GNNs.