# Automated Graph Self-supervised Learning via Multi-teacher Knowledge Distillation

arXiv: https://arxiv.org/abs/2210.02099 · HTML: https://mlsanity.com/paper/2210.02099/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in Graph neural network distillation, one of 12 directions in Knowledge distillation.
- Builds on 35 of its 56 references that are in the map.
- First posted on arXiv 5 Oct 2022.

## Placed in

- Knowledge distillation / [Graph neural network distillation](https://mlsanity.com/distill/graph-neural-network-distillation/)

## Builds on (most cited first, 30 of 35)

- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [Open Graph Benchmark: Datasets for Machine Learning on Graphs](https://mlsanity.com/paper/2005.00687/) (2020)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [Graph Contrastive Learning with Augmentations](https://mlsanity.com/paper/2010.13902/) (2020)
- [Deep Graph Infomax](https://mlsanity.com/paper/1809.10341/) (2018)
- [Pitfalls of Graph Neural Network Evaluation](https://mlsanity.com/paper/1811.05868/) (2018)
- [Strategies for Pre-training Graph Neural Networks](https://mlsanity.com/paper/1905.12265/) (2019)
- [GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks](https://mlsanity.com/paper/1711.02257/) (2017)
- [Contrastive Multi-View Representation Learning on Graphs](https://mlsanity.com/paper/2006.05582/) (2020)
- [Deep Graph Contrastive Representation Learning](https://mlsanity.com/paper/2006.04131/) (2020)
- [Graph Contrastive Learning with Adaptive Augmentation](https://mlsanity.com/paper/2010.14945/) (2020)
- [InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization](https://mlsanity.com/paper/1908.01000/) (2019)
- [Graph Self-Supervised Learning: A Survey](https://mlsanity.com/paper/2103.00111/) (2021)
- [Graph Contrastive Learning Automated](https://mlsanity.com/paper/2106.07594/) (2021)
- [Adversarial Attacks on Graph Neural Networks via Meta Learning](https://mlsanity.com/paper/1902.08412/) (2019)
- [Graph Representation Learning via Graphical Mutual Information Maximization](https://mlsanity.com/paper/2002.01169/) (2020)
- [Self-Supervised Learning of Graph Neural Networks: A Unified Review](https://mlsanity.com/paper/2102.10757/) (2021)
- [SimGRACE: A Simple Framework for Graph Contrastive Learning without Data Augmentation](https://mlsanity.com/paper/2202.03104/) (2022)
- [Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillation](https://mlsanity.com/paper/2110.08727/) (2021)
- [When Does Self-Supervision Help Graph Convolutional Networks?](https://mlsanity.com/paper/2006.09136/) (2020)
- [Rethinking Soft Labels for Knowledge Distillation: A Bias-Variance Tradeoff Perspective](https://mlsanity.com/paper/2102.00650/) (2021)
- [Self-supervised Learning on Graphs: Deep Insights and New Direction](https://mlsanity.com/paper/2006.10141/) (2020)
- [Automated Self-Supervised Learning for Graphs](https://mlsanity.com/paper/2106.05470/) (2021)
- [Contrastive and Generative Graph Convolutional Networks for Graph-based Semi-Supervised Learning](https://mlsanity.com/paper/2009.07111/) (2020)
- [AutoLoss: Automated Loss Function Search in Recommendations](https://mlsanity.com/paper/2106.06713/) (2021)

## Built on by (most cited first, 6 of 6)

- [Re-Dock: Towards Flexible and Realistic Molecular Docking with Diffusion Bridge](https://mlsanity.com/paper/2402.11459/) (2024)
- [Protein 3D Graph Structure Learning for Robust Structure-based Protein Property Prediction](https://mlsanity.com/paper/2310.11466/) (2023)
- [A Survey on Self-Supervised Graph Foundation Models: Knowledge-Based Perspective](https://mlsanity.com/paper/2403.16137/) (2024)
- [Decoupling Weighing and Selecting for Integrating Multiple Graph Pre-training Tasks](https://mlsanity.com/paper/2403.01400/) (2024)
- [Noisy Node Classification by Bi-level Optimization based Multi-teacher Distillation](https://mlsanity.com/paper/2404.17875/) (2024)
- [PaSta: Noisy Node Classification with Partial Label Learning](https://mlsanity.com/paper/2608.25365/) (2026)

## Abstract (arXiv)

Self-supervised learning on graphs has recently achieved remarkable success in graph representation learning. With hundreds of self-supervised pretext tasks proposed over the past few years, the research community has greatly developed, and the key is no longer to design more powerful but complex pretext tasks, but to make more effective use of those already on hand. This paper studies the problem of how to automatically, adaptively, and dynamically learn instance-level self-supervised learning strategies for each node from a given pool of pretext tasks. In this paper, we propose a novel multi-teacher knowledge distillation framework for Automated Graph Self-Supervised Learning (AGSSL), which consists of two main branches: (i) Knowledge Extraction: training multiple teachers with different pretext tasks, so as to extract different levels of knowledge with different inductive biases; (ii) Knowledge Integration: integrating different levels of knowledge and distilling them into the student model. Without simply treating different teachers as equally important, we provide a provable theoretical guideline for how to integrate the knowledge of different teachers, i.e., the integrated teacher probability should be close to the true Bayesian class-probability. To approach the theoretical optimum in practice, two adaptive knowledge integration strategies are proposed to construct a relatively "good" integrated teacher. Extensive experiments on eight datasets show that AGSSL can benefit from multiple pretext tasks, outperforming the corresponding individual tasks; by combining a few simple but classical pretext tasks, the resulting performance is comparable to other leading counterparts.