mlsanityData to 14 Oct 2026

FlyKD: Graph Knowledge Distillation on the Fly with Curriculum Learning

arXiv 2403.10807 ↗First posted 16 Mar 2024cs.LGcited by 0

On the map: Knowledge distillation → Graph neural network distillation

Abstract (arXiv)

Knowledge Distillation (KD) aims to transfer a more capable teacher model's knowledge to a lighter student model in order to improve the efficiency of the model, making it faster and more deployable. However, the student model's optimization process over the noisy pseudo labels (generated by the teacher model) is tricky and the amount of pseudo labels one can generate is limited due to Out of Memory (OOM) error. In this paper, we propose FlyKD (Knowledge Distillation on the Fly) which enables the generation of virtually unlimited number of pseudo labels, coupled with Curriculum Learning that greatly alleviates the optimization process over the noisy pseudo labels. Empirically, we observe that FlyKD outperforms vanilla KD and the renown Local Structure Preserving Graph Convolutional Network (LSPGCN). Lastly, with the success of Curriculum Learning, we shed light on a new research direction of improving optimization over noisy pseudo labels.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Graph neural network distillation, one of 12 directions in Knowledge distillation. Builds on 10 of its 17 references that are in the map. First posted on arXiv 16 Mar 2024.

Builds on

10 of 17 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Knowledge Distillation on Graphs: A Survey1 Feb

Other references · 9

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  3. 2017Graph Attention Networkscited by 8,923
  4. 2018How Powerful are Graph Neural Networks?cited by 4,393
  5. 2017Modeling Relational Data with Graph Convolutional Networkscited by 1,694
  6. 2018Graph Convolutional Neural Networks for Web-Scale Recommender Systemscited by 1,138
  7. 2014Embedding Entities and Relations for Learning and Inference in Knowledge Basescited by 932
  8. 2021How Attentive are Graph Attention Networks?cited by 835
  9. 2021Weight Initialization Techniques for Deep Learning Algorithms in Remote Sensing: Recent Trends and Future Perspectivescited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Adversarial Curriculum Graph-Free Knowledge Distillation for Graph Neural Networks
  2. 2023Frameless Graph Knowledge Distillation
  3. 2023Graph-based Knowledge Distillation: A survey and experimental evaluation
  4. 2021Extract the Knowledge of Graph Neural Networks and Go Beyond it: An Effective Knowledge Distillation Framework
  5. 2021Graph-Free Knowledge Distillation for Graph Neural Networks
  6. 2024A Teacher-Free Graph Knowledge Distillation Framework with Dual Self-Distillation
  7. 2025Efficient Graph Knowledge Distillation from GNNs to Kolmogorov--Arnold Networks via Self-Attention Dynamic Sampling
  8. 2025Enhancing Graph Neural Networks: A Mutual Learning Approach