mlsanityData to 14 Oct 2026

Mixture of Weak & Strong Experts on Graphs

arXiv 2311.05185 ↗First posted 9 Nov 2023cs.LG cs.AIcited by 7

Not on the map. Why?

Abstract (arXiv)

Realistic graphs contain both (1) rich self-features of nodes and (2) informative structures of neighborhoods, jointly handled by a Graph Neural Network (GNN) in the typical setup. We propose to decouple the two modalities by Mixture of weak and strong experts (Mowst), where the weak expert is a light-weight Multi-layer Perceptron (MLP), and the strong expert is an off-the-shelf GNN. To adapt the experts' collaboration to different target nodes, we propose a "confidence" mechanism based on the dispersion of the weak expert's prediction logits. The strong expert is conditionally activated in the low-confidence region when either the node's classification relies on neighborhood information, or the weak expert has low model quality. We reveal interesting training dynamics by analyzing the influence of the confidence function on loss: our training algorithm encourages the specialization of each expert by effectively generating soft splitting of the graph. In addition, our "confidence" design imposes a desirable bias toward the strong expert to benefit from GNN's better generalization capability. Mowst is easy to optimize and achieves strong expressive power, with a computation cost comparable to a single GNN. Empirically, Mowst on 4 backbone GNN architectures show significant accuracy improvement on 6 standard node classification benchmarks, including both homophilous and heterophilous graphs (https://github.com/facebookresearch/mowst-gnn).

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 46 of its 64 references that are in the map. First posted on arXiv 9 Nov 2023.

Builds on

46 of 64 references.

References · 46

  1. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  2. 2017Graph Attention Networkscited by 8,923
  3. 2017Inductive Representation Learning on Large Graphscited by 6,320
  4. 2018How Powerful are Graph Neural Networks?cited by 4,393
  5. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  6. 2019A Comprehensive Survey on Graph Neural Networkscited by 2,919
  7. 2021Convex optimizationcited by 2,515
  8. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,163
  9. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  10. 2018Graph Neural Networks: A Review of Methods and Applicationscited by 1,576
  11. 2018Graph Convolutional Neural Networks for Web-Scale Recommender Systemscited by 1,138
  12. 2018Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learningcited by 1,102
34 more
  1. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  2. 2020Simple and Deep Graph Convolutional Networkscited by 860
  3. 2021How Attentive are Graph Attention Networks?cited by 835
  4. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  5. 2020On the Bottleneck of Graph Neural Networks and its Practical Implicationscited by 631
  6. 2020Adaptive Universal Generalized PageRank Graph Neural Networkcited by 595
  7. 2018FastGCN: Fast Learning with Graph Convolutional Networks via Importance Samplingcited by 584
  8. 2019Measuring and Relieving the Over-smoothing Problem for Graph Neural Networks from the Topological Viewcited by 545
  9. 2021Understanding over-squashing and bottlenecks on graphs via curvaturecited by 478
  10. 2019MixHop: Higher-Order Graph Convolutional Architectures via Sparsified Neighborhood Mixingcited by 443
  11. 2023A critical look at the evaluation of GNNs under heterophily: Are we really making progress?cited by 343
  12. 2017The Expressive Power of Neural Networks: A View from the Widthcited by 320
  13. 2021Large Scale Learning on Non-Homophilous Graphs: New Benchmarks and Strong Simple Methodscited by 311
  14. 2020SIGN: Scalable Inception Graph Neural Networkscited by 284
  15. 2023From Sparse to Soft Mixtures of Expertscited by 248
  16. 2020Graph Neural Networks with Heterophilycited by 202
  17. 2021From Stars to Subgraphs: Uplifting Any GNN with Local Structure Awarenesscited by 155
  18. 2021Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillationcited by 152
  19. 2022A Review of Sparse Expert Models in Deep Learningcited by 148
  20. 2020AM-GCN: Adaptive Multi-channel Graph Convolutional Networkscited by 121
  21. 2021Breaking the Limit of Graph Neural Networks by Improving the Assortativity of Graphs with Local Mixing Patternscited by 95
  22. 2022Decoupling the Depth and Scope of Graph Neural Networkscited by 94
  23. 2021GNNAutoScale: Scalable and Expressive Graph Neural Networks via Historical Embeddingscited by 90
  24. 2023On the Expressive Power of Geometric Graph Neural Networkscited by 90
  25. 2023Graph Mixture of Experts: Learning on Large-Scale Graphs with Explicit Diversity Modelingcited by 64
  26. 2021Twitch Gamers: a Dataset for Evaluating Proximity Preserving and Structural Role-based Node Embeddingscited by 49
  27. 2021Node Dependent Local Smoothing for Scalable Graph Learningcited by 45
  28. 2022Graph Neural Networks are Inherently Good Generalizers: Insights by Bridging GNNs and MLPscited by 42
  29. 2019AdaGCN: Adaboosting Graph Convolutional Networks into Deep Modelscited by 37
  30. 2023LMC: Fast Training of GNNs via Subgraph Sampling with Provable Convergencecited by 26
  31. 2023Finding the Missing-half: Graph Complementary Learning for Homophily-prone and Heterophily-prone Graphscited by 25
  32. 2022Influence-Based Mini-Batching for Graph Neural Networkscited by 15
  33. 2019Disciplined Quasiconvex Programmingcited by 8
  34. 2022Understanding Political Polarization via Jointly Modeling Users, Connections and Multimodal Contents on Heterogeneous Graphscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 2; 2026: 2.

7 papers.

  1. 2024One Model for One Graph: A New Perspective for Pretraining with Cross-domain Graphscited by 12
  2. 2024Interpretable Cascading Mixture-of-Experts for Urban Traffic Congestion Predictioncited by 7
  3. 2024KnowGraph: Knowledge-Enabled Anomaly Detection via Logical Reasoning on Graph Datacited by 4
  4. 2025Graph Few-Shot Learning via Adaptive Spectrum Experts and Cross-Set Distribution Calibrationcited by 3
  5. 2025Out-of-Distribution Graph Models Mergingcited by 0
  6. 2026Zero-shot Generalizable Graph Anomaly Detection with Mixture of Riemannian Expertscited by 0
  7. 2026Closed-Form Node Classification with Exact Graph Unlearningcited by 0