# Online Adversarial Knowledge Distillation for Graph Neural Networks

arXiv: https://arxiv.org/abs/2112.13966 · HTML: https://mlsanity.com/paper/2112.13966/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Graph neural network distillation, one of 12 directions in Knowledge distillation.
- First posted on arXiv 28 Dec 2021.

## Placed in

- Knowledge distillation / [Graph neural network distillation](https://mlsanity.com/distill/graph-neural-network-distillation/)

## Built on by (most cited first, 2 of 2)

- [Deep Graph Reprogramming](https://mlsanity.com/paper/2304.14593/) (2023)
- [Dynamic Data-Free Knowledge Distillation by Easy-to-Hard Learning Strategy](https://mlsanity.com/paper/2208.13648/) (2022)

## Abstract (arXiv)

Knowledge distillation, a technique recently gaining popularity for enhancing model generalization in Convolutional Neural Networks (CNNs), operates under the assumption that both teacher and student models are trained on identical data distributions. However, its effect on Graph Neural Networks (GNNs) is less than satisfactory since the graph topology and node attributes are prone to evolve, thereby leading to the issue of distribution shift. In this paper, we tackle this challenge by simultaneously training a group of graph neural networks in an online distillation fashion, where the group knowledge plays a role as a dynamic virtual teacher and the structure changes in graph neural networks are effectively captured. To improve the distillation performance, two types of knowledge are transferred among the students to enhance each other: local knowledge reflecting information in the graph topology and node attributes, and global knowledge reflecting the prediction over classes. We transfer the global knowledge with KL-divergence as the vanilla knowledge distillation does, while exploiting the complicated structure of the local knowledge with an efficient adversarial cyclic learning framework. Extensive experiments verified the effectiveness of our proposed online adversarial distillation approach. The code is published at https://github.com/wangz3066/OnlineDistillGCN.