Mixture of Scope Experts at Test: Generalizing Deeper Graph Neural Networks with Shallow Variants
Not on the map. Why?
Abstract (arXiv)
Heterophilous graphs, where dissimilar nodes tend to connect, pose a challenge for graph neural networks (GNNs). Increasing the GNN depth can expand the scope (i.e., receptive field), potentially finding homophily from the higher-order neighborhoods. However, GNNs suffer from performance degradation as depth increases. Despite having better expressivity, state-of-the-art deeper GNNs achieve only marginal improvements compared to their shallow variants. Through theoretical and empirical analysis, we systematically demonstrate a shift in GNN generalization preferences across nodes with different homophily levels as depth increases. This creates a disparity in generalization patterns between GNN models with varying depth. Based on these findings, we propose to improve deeper GNN generalization while maintaining high expressivity by Mixture of scope experts at test (Moscat). Experimental results show that Moscat works flexibly with various GNNs across a wide range of datasets while significantly improving accuracy. Our code is available at (https://github.com/Hydrapse/moscat).
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 67 of its 83 references that are in the map. First posted on arXiv 11 Sep 2024.
Builds on
67 of 83 references.
References · 67
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
- 2017Graph Attention Networkscited by 8,854
- 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
- 2017Inductive Representation Learning on Large Graphscited by 6,284
- 2016Layer Normalizationcited by 5,579
- 2018How Powerful are Graph Neural Networks?cited by 4,355
- 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
- 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
- 2019Simplifying Graph Convolutional Networkscited by 1,470
- 2018Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learningcited by 1,099
- 2018Predict then Propagate: Graph Neural Networks meet Personalized PageRankcited by 987
38 more
- 2018Weisfeiler and Leman Go Neural: Higher-order Graph Neural Networkscited by 878
- 2018Representation Learning on Graphs with Jumping Knowledge Networkscited by 875
- 2020Simple and Deep Graph Convolutional Networkscited by 857
- 2020Geom-GCN: Geometric Graph Convolutional Networkscited by 821
- 2022Recipe for a General, Powerful, Scalable Graph Transformercited by 702
- 2019GraphSAINT: Graph Sampling Based Inductive Learning Methodcited by 618
- 2020Adaptive Universal Generalized PageRank Graph Neural Networkcited by 592
- 2019DeepGCNs: Can GCNs Go as Deep as CNNs?cited by 527
- 2020Principal Neighbourhood Aggregation for Graph Netscited by 477
- 2019MixHop: Higher-Order Graph Convolutional Architectures via Sparsified Neighborhood Mixingcited by 442
- 2023A critical look at the evaluation of GNNs under heterophily: Are we really making progress?cited by 332
- 2021Large Scale Learning on Non-Homophilous Graphs: New Benchmarks and Strong Simple Methodscited by 309
- 2020SIGN: Scalable Inception Graph Neural Networkscited by 283
- 2021Graph Neural Network for Traffic Forecasting: A Surveycited by 260
- 2020DeeperGCN: All You Need to Train Deeper GCNscited by 249
- 2020Can Graph Neural Networks Count Substructures?cited by 235
- 2019What graph neural networks cannot learn: depth vs widthcited by 195
- 2021Two Sides of the Same Coin: Heterophily and Oversmoothing in Graph Convolutional Neural Networkscited by 182
- 2022Finding Global Homophily in Graph Neural Networks When Meeting Heterophilycited by 179
- 2020Scaling Graph Neural Networks with Approximate PageRankcited by 174
- 2022Revisiting Heterophily For Graph Neural Networkscited by 169
- 2021A Survey of Graph Neural Networks for Recommender Systems: Challenges, Methods, and Directionscited by 144
- 2023SGFormer: Simplifying and Empowering Transformers for Large-Graph Representationscited by 131
- 2022Decoupling the Depth and Scope of Graph Neural Networkscited by 94
- 2021Be Confident! Towards Trustworthy Graph Neural Networks via Confidence Calibrationcited by 91
- 2023Edge Directionality Improves Learning on Heterophilic Graphscited by 88
- 2020Scalable Graph Neural Networks via Bidirectional Propagationcited by 87
- 2024Classic GNNs are Strong Baselines: Reassessing GNNs for Node Classificationcited by 83
- 2022Graph Attention Multi-Layer Perceptroncited by 82
- 2021Subgroup Generalization and Fairness of Graph Neural Networkscited by 69
- 2018GeniePath: Graph Neural Networks with Adaptive Receptive Pathscited by 66
- 2020Tackling Over-Smoothing for General Graph Convolutional Networkscited by 66
- 2024Polynormer: Polynomial-Expressive Graph Transformer in Linear Timecited by 64
- 2023Graph Mixture of Experts: Learning on Large-Scale Graphs with Explicit Diversity Modelingcited by 63
- 2022Characterizing Graph Datasets for Node Classification: Homophily-Heterophily Dichotomy and Beyondcited by 60
- 2021On Provable Benefits of Depth in Training Graph Convolutional Networkscited by 58
- 2020Revisiting Over-smoothing in Deep GCNscited by 55
- 2022Gradient Gating for Deep Multi-Rate Learning on Graphscited by 49
50 most cited of 67 All 67 (JSON)
Built on by
Citing papers per year: 2026: 1.
1 paper.