# FreeKD: Free-direction Knowledge Distillation for Graph Neural Networks

arXiv: https://arxiv.org/abs/2206.06561 · HTML: https://mlsanity.com/paper/2206.06561/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in Graph neural network distillation, one of 12 directions in Knowledge distillation.
- Builds on 31 of its 48 references that are in the map.
- First posted on arXiv 14 Jun 2022.

## Placed in

- Knowledge distillation / [Graph neural network distillation](https://mlsanity.com/distill/graph-neural-network-distillation/)

## Builds on (most cited first, 30 of 31)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actor](https://mlsanity.com/paper/1801.01290/) (2018)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [Simplifying Graph Convolutional Networks](https://mlsanity.com/paper/1902.07153/) (2019)
- [Predict then Propagate: Graph Neural Networks meet Personalized PageRank](https://mlsanity.com/paper/1810.05997/) (2018)
- [Simple and Deep Graph Convolutional Networks](https://mlsanity.com/paper/2007.02133/) (2020)
- [Geom-GCN: Geometric Graph Convolutional Networks](https://mlsanity.com/paper/2002.05287/) (2020)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [FastGCN: Fast Learning with Graph Convolutional Networks via Importance Sampling](https://mlsanity.com/paper/1801.10247/) (2018)
- [Improved Knowledge Distillation via Teacher Assistant](https://mlsanity.com/paper/1902.03393/) (2019)
- [Multi-scale Attributed Node Embedding](https://mlsanity.com/paper/1909.13021/) (2019)
- [Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks](https://mlsanity.com/paper/1905.07953/) (2019)
- [A Brief Survey of Deep Reinforcement Learning](https://mlsanity.com/paper/1708.05866/) (2017)
- [Adaptive Sampling Towards Fast Graph Representation Learning](https://mlsanity.com/paper/1809.05343/) (2018)
- [Distilling Knowledge from Graph Convolutional Networks](https://mlsanity.com/paper/2003.10477/) (2020)
- [Extract the Knowledge of Graph Neural Networks and Go Beyond it: An Effective Knowledge Distillation Framework](https://mlsanity.com/paper/2103.02885/) (2021)
- [Reinforced Multi-Teacher Selection for Knowledge Distillation](https://mlsanity.com/paper/2012.06048/) (2020)
- [Topology-Imbalance Learning for Semi-Supervised Node Classification](https://mlsanity.com/paper/2110.04099/) (2021)
- [Policy-GNN: Aggregation Optimization for Graph Neural Networks](https://mlsanity.com/paper/2006.15097/) (2020)
- [Graph-Free Knowledge Distillation for Graph Neural Networks](https://mlsanity.com/paper/2105.07519/) (2021)
- [Recurrent Attentional Reinforcement Learning for Multi-label Image Recognition](https://mlsanity.com/paper/1712.07465/) (2017)
- [Multi-Label Image Classification via Knowledge Distillation from Weakly-Supervised Detection](https://mlsanity.com/paper/1809.05884/) (2018)
- [Improving Multi-Task Deep Neural Networks via Knowledge Distillation for Natural Language Understanding](https://mlsanity.com/paper/1904.09482/) (2019)
- [On Self-Distilling Graph Neural Network](https://mlsanity.com/paper/2011.02255/) (2020)
- [Sequential Anomaly Detection using Inverse Reinforcement Learning](https://mlsanity.com/paper/2004.10398/) (2020)
- [ROD: Reception-aware Online Distillation for Sparse Graphs](https://mlsanity.com/paper/2107.11789/) (2021)

## Built on by (most cited first, 23 of 23)

- [Knowledge Distillation on Graphs: A Survey](https://mlsanity.com/paper/2302.00219/) (2023)
- [Teacher-Student Architecture for Knowledge Distillation: A Survey](https://mlsanity.com/paper/2308.04268/) (2023)
- [Quantifying the Knowledge in GNNs for Reliable Distillation into MLPs](https://mlsanity.com/paper/2306.05628/) (2023)
- [Deep Graph Reprogramming](https://mlsanity.com/paper/2304.14593/) (2023)
- [Boosting Graph Neural Networks via Adaptive Knowledge Distillation](https://mlsanity.com/paper/2210.05920/) (2022)
- [Towards Open Temporal Graph Neural Networks](https://mlsanity.com/paper/2303.15015/) (2023)
- [On the Road to Portability: Compressing End-to-End Motion Planner for Autonomous Driving](https://mlsanity.com/paper/2403.01238/) (2024)
- [Robust Knowledge Adaptation for Dynamic Graph Neural Networks](https://mlsanity.com/paper/2207.10839/) (2022)
- [A Teacher-Free Graph Knowledge Distillation Framework with Dual Self-Distillation](https://mlsanity.com/paper/2403.03483/) (2024)
- [MuGSI: Distilling GNNs with Multi-Granularity Structural Information for Graph Classification](https://mlsanity.com/paper/2406.19832/) (2024)
- [TINED: GNNs-to-MLPs by Teacher Injection and Dirichlet Energy Distillation](https://mlsanity.com/paper/2412.11180/) (2024)
- [Teach Harder, Learn Poorer: Rethinking Hard Sample Distillation for GNN-to-MLP Knowledge Distillation](https://mlsanity.com/paper/2407.14768/) (2024)
- [UGMAE: A Unified Framework for Graph Masked Autoencoders](https://mlsanity.com/paper/2402.08023/) (2024)
- [Knowledge-Distilled Graph Neural Networks for Personalized Epileptic Seizure Detection](https://mlsanity.com/paper/2304.06038/) (2023)
- [Shared Growth of Graph Neural Networks via Prompted Free-direction Knowledge Distillation](https://mlsanity.com/paper/2307.00534/) (2023)
- [Pruned Traffic Trees: Native Semantic Compression with a Protocol-Structured Model Family for Encrypted Traffic Classification](https://mlsanity.com/paper/2608.21874/) (2026)
- [E2GNN: Efficient Graph Neural Network Ensembles for Semi-Supervised Classification](https://mlsanity.com/paper/2405.03401/) (2024)
- [Adaptive Teaching with Shared Classifier for Knowledge Distillation](https://mlsanity.com/paper/2406.08528/) (2024)
- [Constraint Multi-class Positive and Unlabeled Learning for Distantly Supervised Named Entity Recognition](https://mlsanity.com/paper/2504.04963/) (2025)
- [Toward Data-centric Directed Graph Learning: An Entropy-driven Approach](https://mlsanity.com/paper/2505.00983/) (2025)
- [Efficient Graph Knowledge Distillation from GNNs to Kolmogorov--Arnold Networks via Self-Attention Dynamic Sampling](https://mlsanity.com/paper/2509.00560/) (2025)
- [Model-to-Data Distillation for Graph Neural Networks](https://mlsanity.com/paper/2605.06814/) (2026)
- [More Domains, No Gains? FLO: Fast-Slow Reciprocal Learning for Multi-Domain Graph Pre-training](https://mlsanity.com/paper/2610.14748/) (2026)

## Abstract (arXiv)

Knowledge distillation (KD) has demonstrated its effectiveness to boost the performance of graph neural networks (GNNs), where its goal is to distill knowledge from a deeper teacher GNN into a shallower student GNN. However, it is actually difficult to train a satisfactory teacher GNN due to the well-known over-parametrized and over-smoothing issues, leading to invalid knowledge transfer in practical applications. In this paper, we propose the first Free-direction Knowledge Distillation framework via Reinforcement learning for GNNs, called FreeKD, which is no longer required to provide a deeper well-optimized teacher GNN. The core idea of our work is to collaboratively build two shallower GNNs in an effort to exchange knowledge between them via reinforcement learning in a hierarchical way. As we observe that one typical GNN model often has better and worse performances at different nodes during training, we devise a dynamic and free-direction knowledge transfer strategy that consists of two levels of actions: 1) node-level action determines the directions of knowledge transfer between the corresponding nodes of two networks; and then 2) structure-level action determines which of the local structures generated by the node-level actions to be propagated. In essence, our FreeKD is a general and principled framework which can be naturally compatible with GNNs of different architectures. Extensive experiments on five benchmark datasets demonstrate our FreeKD outperforms two base GNNs in a large margin, and shows its efficacy to various GNNs. More surprisingly, our FreeKD has comparable or even better performance than traditional KD algorithms that distill knowledge from a deeper and stronger teacher GNN.