# Adaptive Transfer Learning on Graph Neural Networks

arXiv: https://arxiv.org/abs/2107.08765 · HTML: https://mlsanity.com/paper/2107.08765/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 37 references that are in the map.
- First posted on arXiv 19 Jul 2021.

## Builds on (most cited first, 30 of 31)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Variational Graph Auto-Encoders](https://mlsanity.com/paper/1611.07308/) (2016)
- [Multi-Task Learning Using Uncertainty to Weigh Losses for Scene Geometry and Semantics](https://mlsanity.com/paper/1705.07115/) (2017)
- [Graph Convolutional Neural Networks for Web-Scale Recommender Systems](https://mlsanity.com/paper/1806.01973/) (2018)
- [Deep Graph Infomax](https://mlsanity.com/paper/1809.10341/) (2018)
- [Spectral Networks and Locally Connected Networks on Graphs](https://mlsanity.com/paper/1312.6203/) (2013)
- [Strategies for Pre-training Graph Neural Networks](https://mlsanity.com/paper/1905.12265/) (2019)
- [GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networks](https://mlsanity.com/paper/1711.02257/) (2017)
- [Link Prediction Based on Graph Neural Networks](https://mlsanity.com/paper/1802.09691/) (2018)
- [Hierarchical Graph Representation Learning with Differentiable Pooling](https://mlsanity.com/paper/1806.08804/) (2018)
- [Graph Neural Networks for Social Recommendation](https://mlsanity.com/paper/1902.07243/) (2019)
- [Heterogeneous Graph Transformer](https://mlsanity.com/paper/2003.01332/) (2020)
- [End-to-End Multi-Task Learning with Attention](https://mlsanity.com/paper/1803.10704/) (2018)
- [InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximization](https://mlsanity.com/paper/1908.01000/) (2019)
- [Multi-Task Deep Neural Networks for Natural Language Understanding](https://mlsanity.com/paper/1901.11504/) (2019)
- [GPT-GNN: Generative Pre-Training of Graph Neural Networks](https://mlsanity.com/paper/2006.15437/) (2020)
- [RippleNet: Propagating User Preferences on the Knowledge Graph for Recommender Systems](https://mlsanity.com/paper/1803.03467/) (2018)
- [When Does Self-Supervision Help Graph Convolutional Networks?](https://mlsanity.com/paper/2006.09136/) (2020)
- [Multi-Stage Self-Supervised Learning for Graph Convolutional Networks on Graphs with Few Labels](https://mlsanity.com/paper/1902.11038/) (2019)
- [Self-supervised Learning on Graphs: Deep Insights and New Direction](https://mlsanity.com/paper/2006.10141/) (2020)
- [Knowledge-aware Graph Neural Networks with Label Smoothness Regularization for Recommender Systems](https://mlsanity.com/paper/1905.04413/) (2019)
- [Adapting Auxiliary Losses Using Gradient Similarity](https://mlsanity.com/paper/1812.02224/) (2018)
- [Self-Supervised Generalisation with Meta Auxiliary Learning](https://mlsanity.com/paper/1901.08933/) (2019)
- [An Embarrassingly Simple Approach for Transfer Learning from Pretrained Language Models](https://mlsanity.com/paper/1902.10547/) (2019)
- [Self-Supervised Graph Representation Learning via Global Context Prediction](https://mlsanity.com/paper/2003.01604/) (2020)

## Built on by (most cited first, 30 of 31)

- [Hierarchical Graph Transformer with Adaptive Node Sampling](https://mlsanity.com/paper/2210.03930/) (2022)
- [Non-IID Transfer Learning on Graphs](https://mlsanity.com/paper/2212.08174/) (2022)
- [GeoLayoutLM: Geometric Pre-training for Visual Information Extraction](https://mlsanity.com/paper/2304.10759/) (2023)
- [Graph-Aware Language Model Pre-Training on a Large Graph Corpus Can Help Multiple Graph Applications](https://mlsanity.com/paper/2306.02592/) (2023)
- [A Survey of Pretraining on Graphs: Taxonomy, Methods, and Applications](https://mlsanity.com/paper/2202.07893/) (2022)
- [GraphLoRA: Structure-Aware Contrastive Low-Rank Adaptation for Cross-Graph Transfer Learning](https://mlsanity.com/paper/2409.16670/) (2024)
- [MentorGNN: Deriving Curriculum for Pre-Training GNNs](https://mlsanity.com/paper/2208.09905/) (2022)
- [When to Pre-Train Graph Neural Networks? From Data Generation Perspective!](https://mlsanity.com/paper/2303.16458/) (2023)
- [Subgraph Pooling: Tackling Negative Transfer on Graphs](https://mlsanity.com/paper/2402.08907/) (2024)
- [Search to Fine-tune Pre-trained Graph Neural Networks for Graph-level Tasks](https://mlsanity.com/paper/2308.06960/) (2023)
- [Fine-tuning Graph Neural Networks by Preserving Graph Generative Patterns](https://mlsanity.com/paper/2312.13583/) (2023)
- [Subgraph-level Universal Prompt Tuning](https://mlsanity.com/paper/2402.10380/) (2024)
- [A Survey on Self-Supervised Graph Foundation Models: Knowledge-Based Perspective](https://mlsanity.com/paper/2403.16137/) (2024)
- [RELIEF: Reinforcement Learning Empowered Graph Feature Prompt Tuning](https://mlsanity.com/paper/2408.03195/) (2024)
- [Use of Graph Neural Networks in Aiding Defensive Cyber Operations](https://mlsanity.com/paper/2401.05680/) (2024)
- [Decoupling Weighing and Selecting for Integrating Multiple Graph Pre-training Tasks](https://mlsanity.com/paper/2403.01400/) (2024)
- [Exploring Correlations of Self-Supervised Tasks for Graphs](https://mlsanity.com/paper/2405.04245/) (2024)
- [Towards Continuous Reuse of Graph Models via Holistic Memory Diversification](https://mlsanity.com/paper/2406.07413/) (2024)
- [Automated Graph Self-supervised Learning via Multi-teacher Knowledge Distillation](https://mlsanity.com/paper/2210.02099/) (2022)
- [FedGT: Federated Node Classification with Scalable Graph Transformer](https://mlsanity.com/paper/2401.15203/) (2024)
- [You Only Transfer What You Share: Intersection-Induced Graph Transfer Learning for Link Prediction](https://mlsanity.com/paper/2302.14189/) (2023)
- [Graph Neural Networks for Surfactant Multi-Property Prediction](https://mlsanity.com/paper/2401.01874/) (2024)
- [Edge Graph Intelligence: Reciprocally Empowering Edge Networks with Graph Intelligence](https://mlsanity.com/paper/2407.15320/) (2024)
- [Unify Graph Learning with Text: Unleashing LLM Potentials for Session Search](https://mlsanity.com/paper/2505.14156/) (2025)
- [Learning and Editing Universal Graph Prompt Tuning via Reinforcement Learning](https://mlsanity.com/paper/2512.08763/) (2025)
- [Transfer Learning via Test-Time Neural Networks Aggregation](https://mlsanity.com/paper/2206.13399/) (2022)
- [From Basic to Extra Features: Hypergraph Transformer Pretrain-then-Finetuning for Balanced Clinical Predictions on EHR](https://mlsanity.com/paper/2406.05682/) (2024)
- [Finding Materialized Models for Model Reuse](https://mlsanity.com/paper/2110.06532/) (2021)
- [Toward Improved Generalization: Meta Transfer of Self-supervised Knowledge on Graphs](https://mlsanity.com/paper/2212.08217/) (2022)
- [Enhancing Document Information Analysis with Multi-Task Pre-training: A Robust Approach for Information Extraction in Visually-Rich Documents](https://mlsanity.com/paper/2310.16527/) (2023)

## Abstract (arXiv)

Graph neural networks (GNNs) is widely used to learn a powerful representation of graph-structured data. Recent work demonstrates that transferring knowledge from self-supervised tasks to downstream tasks could further improve graph representation. However, there is an inherent gap between self-supervised tasks and downstream tasks in terms of optimization objective and training data. Conventional pre-training methods may be not effective enough on knowledge transfer since they do not make any adaptation for downstream tasks. To solve such problems, we propose a new transfer learning paradigm on GNNs which could effectively leverage self-supervised tasks as auxiliary tasks to help the target task. Our methods would adaptively select and combine different auxiliary tasks with the target task in the fine-tuning stage. We design an adaptive auxiliary loss weighting model to learn the weights of auxiliary tasks by quantifying the consistency between auxiliary tasks and the target task. In addition, we learn the weighting model through meta-learning. Our methods can be applied to various transfer learning approaches, it performs well not only in multi-task learning but also in pre-training and fine-tuning. Comprehensive experiments on multiple downstream tasks demonstrate that the proposed methods can effectively combine auxiliary tasks with the target task and significantly improve the performance compared to state-of-the-art methods.