# Adversarial Curriculum Graph-Free Knowledge Distillation for Graph Neural Networks

arXiv: https://arxiv.org/abs/2504.00540 · HTML: https://mlsanity.com/paper/2504.00540/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Graph neural network distillation, one of 12 directions in Knowledge distillation.
- Builds on 37 of its 61 references that are in the map.
- First posted on arXiv 1 Apr 2025.

## Placed in

- Knowledge distillation / [Graph neural network distillation](https://mlsanity.com/distill/graph-neural-network-distillation/)

## Builds on (most cited first, 30 of 37)

- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [FitNets: Hints for Thin Deep Nets](https://mlsanity.com/paper/1412.6550/) (2014)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [The Concrete Distribution: A Continuous Relaxation of Discrete Random Variables](https://mlsanity.com/paper/1611.00712/) (2016)
- [Co-teaching: Robust Training of Deep Neural Networks with Extremely Noisy Labels](https://mlsanity.com/paper/1804.06872/) (2018)
- [MentorNet: Learning Data-Driven Curriculum for Very Deep Neural Networks on Corrupted Labels](https://mlsanity.com/paper/1712.05055/) (2017)
- [Training Region-based Object Detectors with Online Hard Example Mining](https://mlsanity.com/paper/1604.03540/) (2016)
- [Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion](https://mlsanity.com/paper/1912.08795/) (2019)
- [Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlooks](https://mlsanity.com/paper/2004.05937/) (2020)
- [On The Power of Curriculum Learning in Training Deep Networks](https://mlsanity.com/paper/1904.03626/) (2019)
- [Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learning](https://mlsanity.com/paper/2203.09249/) (2022)
- [Zero-shot Knowledge Transfer via Adversarial Belief Matching](https://mlsanity.com/paper/1905.09768/) (2019)
- [Curriculum Learning by Transfer Learning: Theory and Experiments with Deep Networks](https://mlsanity.com/paper/1802.03796/) (2018)
- [Curriculum Temperature for Knowledge Distillation](https://mlsanity.com/paper/2211.16231/) (2022)
- [Logit Standardization in Knowledge Distillation](https://mlsanity.com/paper/2403.01427/) (2024)
- [A Comprehensive Survey on Deep Graph Representation Learning](https://mlsanity.com/paper/2304.05055/) (2023)
- [Contrastive Model Inversion for Data-Free Knowledge Distillation](https://mlsanity.com/paper/2105.08584/) (2021)
- [Graph-Free Knowledge Distillation for Graph Neural Networks](https://mlsanity.com/paper/2105.07519/) (2021)
- [Better Teacher Better Student: Dynamic Prior Knowledge for Knowledge Distillation](https://mlsanity.com/paper/2206.06067/) (2022)
- [Robust and Resource-Efficient Data-Free Knowledge Distillation by Generative Pseudo Replay](https://mlsanity.com/paper/2201.03019/) (2022)
- [Momentum Adversarial Distillation: Handling Large Distribution Shifts in Data-Free Knowledge Distillation](https://mlsanity.com/paper/2209.10359/) (2022)
- [Learning to Retain while Acquiring: Combating Distribution-Shift in Adversarial Data-Free Knowledge Distillation](https://mlsanity.com/paper/2302.14290/) (2023)
- [FedTAD: Topology-aware Data-free Knowledge Distillation for Subgraph Federated Learning](https://mlsanity.com/paper/2404.14061/) (2024)
- [The Best of Both Worlds: Accurate Global and Personalized Models through Federated Learning with Data-Free Hyper-Knowledge Distillation](https://mlsanity.com/paper/2301.08968/) (2023)
- [CLNode: Curriculum Learning for Node Classification](https://mlsanity.com/paper/2206.07258/) (2022)
- [Frequency Attention for Knowledge Distillation](https://mlsanity.com/paper/2403.05894/) (2024)

## Abstract (arXiv)

Data-free Knowledge Distillation (DFKD) is a method that constructs pseudo-samples using a generator without real data, and transfers knowledge from a teacher model to a student by enforcing the student to overcome dimensional differences and learn to mimic the teacher's outputs on these pseudo-samples. In recent years, various studies in the vision domain have made notable advancements in this area. However, the varying topological structures and non-grid nature of graph data render the methods from the vision domain ineffective. Building upon prior research into differentiable methods for graph neural networks, we propose a fast and high-quality data-free knowledge distillation approach in this paper. Without compromising distillation quality, the proposed graph-free KD method (ACGKD) significantly reduces the spatial complexity of pseudo-graphs by leveraging the Binary Concrete distribution to model the graph structure and introducing a spatial complexity tuning parameter. This approach enables efficient gradient computation for the graph structure, thereby accelerating the overall distillation process. Additionally, ACGKD eliminates the dimensional ambiguity between the student and teacher models by increasing the student's dimensions and reusing the teacher's classifier. Moreover, it equips graph knowledge distillation with a CL-based strategy to ensure the student learns graph structures progressively. Extensive experiments demonstrate that ACGKD achieves state-of-the-art performance in distilling knowledge from GNNs without training data.