# Confidence May Cheat: Self-Training on Graph Neural Networks under Distribution Shift

arXiv: https://arxiv.org/abs/2201.11349 · HTML: https://mlsanity.com/paper/2201.11349/

## Key facts

- Cited by 38 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 40 references that are in the map.
- First posted on arXiv 27 Jan 2022.

## Builds on (most cited first, 24 of 24)

- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering](https://mlsanity.com/paper/1606.09375/) (2016)
- [Graph Neural Networks: A Review of Methods and Applications](https://mlsanity.com/paper/1812.08434/) (2018)
- [Simplifying Graph Convolutional Networks](https://mlsanity.com/paper/1902.07153/) (2019)
- [Spectral Networks and Locally Connected Networks on Graphs](https://mlsanity.com/paper/1312.6203/) (2013)
- [Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learning](https://mlsanity.com/paper/1801.07606/) (2018)
- [Deep Clustering for Unsupervised Learning of Visual Features](https://mlsanity.com/paper/1807.05520/) (2018)
- [Predict then Propagate: Graph Neural Networks meet Personalized PageRank](https://mlsanity.com/paper/1810.05997/) (2018)
- [DropEdge: Towards Deep Graph Convolutional Networks on Node Classification](https://mlsanity.com/paper/1907.10903/) (2019)
- [Making Deep Neural Networks Robust to Label Noise: a Loss Correction Approach](https://mlsanity.com/paper/1609.03683/) (2016)
- [Deep Learning on Graphs: A Survey](https://mlsanity.com/paper/1812.04202/) (2018)
- [Deep Gaussian Embedding of Graphs: Unsupervised Inductive Learning via Ranking](https://mlsanity.com/paper/1707.03815/) (2017)
- [Revisiting Self-Training for Neural Sequence Generation](https://mlsanity.com/paper/1909.13788/) (2019)
- [Multi-Stage Self-Supervised Learning for Graph Convolutional Networks on Graphs with Few Labels](https://mlsanity.com/paper/1902.11038/) (2019)
- [Be Confident! Towards Trustworthy Graph Neural Networks via Confidence Calibration](https://mlsanity.com/paper/2109.14285/) (2021)
- [Bayesian Graph Neural Networks with Adaptive Connection Sampling](https://mlsanity.com/paper/2006.04064/) (2020)
- [Confidence Scores Make Instance-dependent Label-noise Learning Possible](https://mlsanity.com/paper/2001.03772/) (2020)
- [Black-box α-divergence Minimization](https://mlsanity.com/paper/1511.03243/) (2015)
- [SLADE: A Self-Training Framework For Distance Metric Learning](https://mlsanity.com/paper/2011.10269/) (2020)

## Built on by (most cited first, 30 of 38)

- [Exploring the Potential of Large Language Models (LLMs) in Learning on Graphs](https://mlsanity.com/paper/2307.03393/) (2023)
- [Out-Of-Distribution Generalization on Graphs: A Survey](https://mlsanity.com/paper/2202.07987/) (2022)
- [Empowering Graph Representation Learning with Test-Time Graph Transformation](https://mlsanity.com/paper/2210.03561/) (2022)
- [Demystifying Structural Disparity in Graph Neural Networks: Can One Size Fit All?](https://mlsanity.com/paper/2306.01323/) (2023)
- [Uncertainty in Graph Neural Networks: A Survey](https://mlsanity.com/paper/2403.07185/) (2024)
- [A Survey of Deep Graph Learning under Distribution Shifts: from Graph Out-of-Distribution Generalization to Adaptation](https://mlsanity.com/paper/2410.19265/) (2024)
- [Pave Your Own Path: Graph Gradual Domain Adaptation on Fused Gromov-Wasserstein Geodesics](https://mlsanity.com/paper/2505.12709/) (2025)
- [Toward Robust Graph Semi-Supervised Learning against Extreme Data Scarcity](https://mlsanity.com/paper/2208.12422/) (2022)
- [Deep Insights into Noisy Pseudo Labeling on Graph Data](https://mlsanity.com/paper/2310.01634/) (2023)
- [Conformalized Link Prediction on Graph Neural Networks](https://mlsanity.com/paper/2406.18763/) (2024)
- [USBD: Universal Structural Basis Distillation for Source-Free Graph Domain Adaptation](https://mlsanity.com/paper/2602.08431/) (2026)
- [GALA: Graph Diffusion-based Alignment with Jigsaw for Source-free Domain Adaptation](https://mlsanity.com/paper/2410.16606/) (2024)
- [Finding Heterophilic Neighbors via Confidence-based Subgraph Matching for Semi-supervised Node Classification](https://mlsanity.com/paper/2302.09755/) (2023)
- [Distribution Consistency based Self-Training for Graph Neural Networks with Sparse Labels](https://mlsanity.com/paper/2401.10394/) (2024)
- [Beyond Generalization: A Survey of Out-Of-Distribution Adaptation on Graphs](https://mlsanity.com/paper/2402.11153/) (2024)
- [Mutually-paced Knowledge Distillation for Cross-lingual Temporal Knowledge Graph Reasoning](https://mlsanity.com/paper/2303.14898/) (2023)
- [Cross-Space Adaptive Filter: Integrating Graph Topology and Node Attributes for Alleviating the Over-smoothing Problem](https://mlsanity.com/paper/2401.14876/) (2024)
- [Uncertainty Quantification on Graph Learning: A Survey](https://mlsanity.com/paper/2404.14642/) (2024)
- [IceBerg: Debiased Self-Training for Class-Imbalanced Node Classification](https://mlsanity.com/paper/2502.06280/) (2025)
- [GNN-as-Judge: Unleashing the Power of LLMs for Graph Learning with GNN Feedback](https://mlsanity.com/paper/2604.08553/) (2026)
- [Improving Signed Propagation for Graph Neural Networks in Multi-Class Environments](https://mlsanity.com/paper/2301.08918/) (2023)
- [Addressing the Impact of Localized Training Data in Graph Neural Networks](https://mlsanity.com/paper/2307.12689/) (2023)
- [HC-GST: Heterophily-aware Distribution Consistency based Graph Self-training](https://mlsanity.com/paper/2407.17787/) (2024)
- [BANGS: Game-Theoretic Node Selection for Graph Self-Training](https://mlsanity.com/paper/2410.09348/) (2024)
- [Intent-calibrated Self-training for Answer Selection in Open-domain Dialogues](https://mlsanity.com/paper/2307.06703/) (2023)
- [CueGCL: Cluster-aware Personalized Self-Training for Unsupervised Graph Contrastive Learning](https://mlsanity.com/paper/2311.11073/) (2023)
- [Towards Human-like Perception: Learning Structural Causal Model in Heterogeneous Graph](https://mlsanity.com/paper/2312.05757/) (2023)
- [Data Augmentation for Supervised Graph Outlier Detection via Latent Diffusion Models](https://mlsanity.com/paper/2312.17679/) (2023)
- [Graph Augmentation for Cross Graph Domain Generalization](https://mlsanity.com/paper/2502.18188/) (2025)
- [Does Homophily Help in Robust Test-time Node Classification?](https://mlsanity.com/paper/2510.22289/) (2025)

## Abstract (arXiv)

Graph Convolutional Networks (GCNs) have recently attracted vast interest and achieved state-of-the-art performance on graphs, but its success could typically hinge on careful training with amounts of expensive and time-consuming labeled data. To alleviate labeled data scarcity, self-training methods have been widely adopted on graphs by labeling high-confidence unlabeled nodes and then adding them to the training step. In this line, we empirically make a thorough study for current self-training methods on graphs. Surprisingly, we find that high-confidence unlabeled nodes are not always useful, and even introduce the distribution shift issue between the original labeled dataset and the augmented dataset by self-training, severely hindering the capability of self-training on graphs. To this end, in this paper, we propose a novel Distribution Recovered Graph Self-Training framework (DR-GST), which could recover the distribution of the original labeled dataset. Specifically, we first prove the equality of loss function in self-training framework under the distribution shift case and the population distribution if each pseudo-labeled node is weighted by a proper coefficient. Considering the intractability of the coefficient, we then propose to replace the coefficient with the information gain after observing the same changing trend between them, where information gain is respectively estimated via both dropout variational inference and dropedge variational inference in DR-GST. However, such a weighted loss function will enlarge the impact of incorrect pseudo labels. As a result, we apply the loss correction method to improve the quality of pseudo labels. Both our theoretical analysis and extensive experiments on five benchmark datasets demonstrate the effectiveness of the proposed DR-GST, as well as each well-designed component in DR-GST.