Universal Deep GNNs: Rethinking Residual Connection in GNNs from a Path Decomposition Perspective for Preventing the Over-smoothing
Not on the map. Why?
Abstract (arXiv)
The performance of GNNs degrades as they become deeper due to the over-smoothing. Among all the attempts to prevent over-smoothing, residual connection is one of the promising methods due to its simplicity. However, recent studies have shown that GNNs with residual connections only slightly slow down the degeneration. The reason why residual connections fail in GNNs is still unknown. In this paper, we investigate the forward and backward behavior of GNNs with residual connections from a novel path decomposition perspective. We find that the recursive aggregation of the median length paths from the binomial distribution of residual connection paths dominates output representation, resulting in over-smoothing as GNNs go deeper. Entangled propagation and weight matrices cause gradient smoothing and prevent GNNs with residual connections from optimizing to the identity mapping. Based on these findings, we present a Universal Deep GNNs (UDGNN) framework with cold-start adaptive residual connections (DRIVE) and feedforward modules. Extensive experiments demonstrate the effectiveness of our method, which achieves state-of-the-art results over non-smooth heterophily datasets by simply stacking standard GNNs.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 42 of its 50 references that are in the map. First posted on arXiv 30 May 2022.
Builds on
42 of 50 references.
References · 42
- 2017Attention Is All You Needcited by 60,361
- 2015Deep Residual Learning for Image Recognitioncited by 52,096
- 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
- 2017Graph Attention Networkscited by 8,923
- 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
- 2017Inductive Representation Learning on Large Graphscited by 6,320
- 2016PointNet: Deep Learning on Point Sets for 3D Classification and Segmentationcited by 5,248
- 2019A Comprehensive Survey on Graph Neural Networkscited by 2,919
- 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,163
- 2016Identity Mappings in Deep Residual Networkscited by 1,965
- 2019Simplifying Graph Convolutional Networkscited by 1,477
- 2020LightGCN: Simplifying and Powering Graph Convolution Network for Recommendationcited by 1,139
30 more
- 2018Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learningcited by 1,102
- 2018Predict then Propagate: Graph Neural Networks meet Personalized PageRankcited by 993
- 2018Representation Learning on Graphs with Jumping Knowledge Networkscited by 877
- 2020Simple and Deep Graph Convolutional Networkscited by 860
- 2020Geom-GCN: Geometric Graph Convolutional Networkscited by 826
- 2019DropEdge: Towards Deep Graph Convolutional Networks on Node Classificationcited by 680
- 2020Adaptive Universal Generalized PageRank Graph Neural Networkcited by 595
- 2021MetaFormer Is Actually What You Need for Visioncited by 524
- 2019MixHop: Higher-Order Graph Convolutional Architectures via Sparsified Neighborhood Mixingcited by 443
- 2021Attention is Not All You Need: Pure Attention Loses Rank Doubly Exponentially with Depthcited by 419
- 2021Beyond Low-frequency Information in Graph Convolutional Networkscited by 362
- 2021Large Scale Learning on Non-Homophilous Graphs: New Benchmarks and Strong Simple Methodscited by 311
- 2019PairNorm: Tackling Oversmoothing in GNNscited by 297
- 2020Towards Deeper Graph Neural Networkscited by 296
- 2019Revisiting Graph Neural Networks: All We Have is Low-Pass Filterscited by 261
- 2020DeeperGCN: All You Need to Train Deeper GCNscited by 250
- 2020A Note on Over-Smoothing for Graph Neural Networkscited by 249
- 2020Graph Random Neural Network for Semi-Supervised Learning on Graphscited by 213
- 2020Design Space for Graph Neural Networkscited by 182
- 2021Two Sides of the Same Coin: Heterophily and Oversmoothing in Graph Convolutional Neural Networkscited by 182
- 2018Traffic Graph Convolutional Recurrent Neural Network: A Deep Learning Framework for Network-Scale Traffic Learning and Forecastingcited by 146
- 2020Towards Deeper Graph Neural Networks with Differentiable Group Normalizationcited by 114
- 2021Dirichlet Energy Constrained Learning for Deep Graph Neural Networkscited by 97
- 2019DeepGCNs: Making GCNs Go as Deep as CNNscited by 72
- 2020Tackling Over-Smoothing for General Graph Convolutional Networkscited by 66
- 2021On Provable Benefits of Depth in Training Graph Convolutional Networkscited by 58
- 2018Graph Transformation Policy Network for Chemical Reaction Predictioncited by 56
- 2019Predicting Path Failure In Time-Evolving Graphscited by 54
- 2019Estimating Node Importance in Knowledge Graphs Using Graph Neural Networkscited by 43
- 2021Bag of Tricks for Training Deeper Graph Neural Networks: A Comprehensive Benchmark Studycited by 36
Built on by
Citing papers per year: 2022: 1.
1 paper.