mlsanityData to 7 Oct 2026

Graph Mixture of Experts: Learning on Large-Scale Graphs with Explicit Diversity Modeling

arXiv 2304.02806 ↗First posted 6 Apr 2023cs.LGcited by 63

On the map: Mixture-of-experts → MoE for time series and graphs

Abstract (arXiv)

Graph neural networks (GNNs) have found extensive applications in learning from graph data. However, real-world graphs often possess diverse structures and comprise nodes and edges of varying types. To bolster the generalization capacity of GNNs, it has become customary to augment training graph structures through techniques like graph augmentations and large-scale pre-training on a wider array of graphs. Balancing this diversity while avoiding increased computational costs and the notorious trainability issues of GNNs is crucial. This study introduces the concept of Mixture-of-Experts (MoE) to GNNs, with the aim of augmenting their capacity to adapt to a diverse range of training graph structures, without incurring explosive computational overhead. The proposed Graph Mixture of Experts (GMoE) model empowers individual nodes in the graph to dynamically and adaptively select more general information aggregation experts. These experts are trained to capture distinct subgroups of graph structures and to incorporate information with varying hop sizes, where those with larger hop sizes specialize in gathering information over longer distances. The effectiveness of GMoE is validated through a series of experiments on a diverse set of tasks, including graph, node, and link prediction, using the OGB benchmark. Notably, it enhances ROC-AUC by 1.81% in ogbg-molhiv and by 1.40% in ogbg-molbbbp, when compared to the non-MoE baselines. Our code is publicly available at https://github.com/VITA-Group/Graph-Mixture-of-Experts.

Key facts. Cited by 63 papers in the map (data through 7 Oct 2026). Placed in MoE for time series and graphs, one of 12 directions in Mixture-of-experts. Builds on 56 of its 77 references that are in the map. First posted on arXiv 6 Apr 2023.

Builds on

56 of 77 references.

References · 56

  1. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  2. 2017Graph Attention Networkscited by 8,854
  3. 2017Inductive Representation Learning on Large Graphscited by 6,284
  4. 2018How Powerful are Graph Neural Networks?cited by 4,355
  5. 2017Neural Message Passing for Quantum Chemistrycited by 3,460
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  7. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  8. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  9. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,148
  10. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  11. 2016Variational Graph Auto-Encoderscited by 1,373
  12. 2020Graph Contrastive Learning with Augmentationscited by 1,203
38 more
  1. 2018Deep Graph Infomaxcited by 1,101
  2. 2019Strategies for Pre-training Graph Neural Networkscited by 964
  3. 2020Geom-GCN: Geometric Graph Convolutional Networkscited by 821
  4. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  5. 2020Deep Graph Contrastive Representation Learningcited by 535
  6. 2020Graph Contrastive Learning with Adaptive Augmentationcited by 519
  7. 2020Graph Neural Networks in Recommender Systems: A Surveycited by 493
  8. 2020GCC: Graph Contrastive Coding for Graph Neural Network Pre-Trainingcited by 461
  9. 2022GraphMAE: Self-Supervised Masked Graph Autoencoderscited by 443
  10. 2019MixHop: Higher-Order Graph Convolutional Architectures via Sparsified Neighborhood Mixingcited by 442
  11. 2019InfoGraph: Unsupervised and Semi-supervised Graph-Level Representation Learning via Mutual Information Maximizationcited by 441
  12. 2018GraphRNN: Generating Realistic Graphs with Deep Auto-regressive Modelscited by 406
  13. 2018Graph Convolutional Policy Network for Goal-Directed Molecular Graph Generationcited by 363
  14. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  15. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  16. 2020GPT-GNN: Generative Pre-Training of Graph Neural Networkscited by 302
  17. 2021Graph Contrastive Learning Automatedcited by 287
  18. 2019Adversarial Examples Improve Image Recognitioncited by 283
  19. 2021Graph Neural Network for Traffic Forecasting: A Surveycited by 260
  20. 2020DeeperGCN: All You Need to Train Deeper GCNscited by 249
  21. 2022Unified Scaling Laws for Routed Language Modelscited by 249
  22. 2021Hash Layers For Large Sparse Modelscited by 236
  23. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202
  24. 2022G-Mixup: Graph Data Augmentation for Graph Classificationcited by 145
  25. 2021From Canonical Correlation Analysis to Self-supervised Graph Neural Networkscited by 138
  26. 2020When Does Self-Supervision Help Graph Convolutional Networks?cited by 121
  27. 2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designcited by 97
  28. 2019Symmetric Graph Convolutional Autoencoder for Unsupervised Graph Representation Learningcited by 94
  29. 2021AugMax: Adversarial Composition of Random Augmentations for Robust Trainingcited by 89
  30. 2021Characterizing signal propagation to close the performance gap in unnormalized ResNetscited by 87
  31. 2022Sparse Mixture-of-Experts are Domain Generalizable Learnerscited by 85
  32. 2020Large-scale Robust Deep AUC Maximization: A New Surrogate Loss and Empirical Studies on Medical Image Classificationcited by 84
  33. 2022Uni-Perceiver-MoE: Learning Sparse Generalist Models with Conditional MoEscited by 58
  34. 2018N-GCN: Multi-scale Graph Convolution for Semi-supervised Node Classificationcited by 54
  35. 2021Local Augmentation for Graph Neural Networkscited by 52
  36. 2019Graph Attention Auto-Encoderscited by 41
  37. 2017Learning Graph Representations with Embedding Propagationcited by 38
  38. 2021Pooling Architecture Search for Graph Classificationcited by 36

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 15; 2025: 29; 2026: 15.

63 papers.

Sort:

By area:

  1. 2023Graph Foundation Models: Concepts, Opportunities and Challengescited by 101
  2. 2025UniGraph2: Learning a Unified Embedding Space to Bind Multimodal Graphscited by 47
  3. 2024Mix-of-Granularity: Optimize the Chunking Granularity for Retrieval-Augmented Generationcited by 23
  4. 2023GraphMETRO: Mitigating Complex Graph Distribution Shifts via Mixture of Aligned Expertscited by 19
  5. 2024Graph Mixture of Experts and Memory-augmented Routers for Multivariate Time Series Anomaly Detectioncited by 19
  6. 2024Graph Sparsification via Mixture of Graphscited by 17
  7. 2024Node-wise Filtering in Graph Neural Networks: A Mixture of Experts Approachcited by 17
  8. 2024Mixture of In-Context Experts Enhance LLMs' Long Context Awarenesscited by 17
  9. 2025GRAVER: Generative Graph Vocabularies for Robust Graph Foundation Models Fine-tuningcited by 13
  10. 2024One Model for One Graph: A New Perspective for Pretraining with Cross-domain Graphscited by 12
  11. 2024Mixture of Link Predictors on Graphscited by 10
  12. 2024Hierarchical Mixture of Experts: Generalizable Learning for High-Level Synthesiscited by 9
38 more
  1. 2023Mixture of Weak & Strong Experts on Graphscited by 7
  2. 2023PGODE: Towards High-quality System Dynamics Modelingcited by 7
  3. 2024Interpretable Cascading Mixture-of-Experts for Urban Traffic Congestion Predictioncited by 7
  4. 2024DA-MoE: Addressing Depth-Sensitivity in Graph-Level Analysis through Mixture of Expertscited by 7
  5. 2025Cooperation of Experts: Fusing Heterogeneous Information with Large Margincited by 6
  6. 2025ToVE: Efficient Vision-Language Learning via Knowledge Transfer from Vision Expertscited by 5
  7. 2025Nonparametric Teaching for Graph Property Learnerscited by 5
  8. 2025Mixture of Length and Pruning Experts for Knowledge Graphs Reasoningcited by 5
  9. 2024KnowGraph: Knowledge-Enabled Anomaly Detection via Logical Reasoning on Graph Datacited by 4
  10. 2025ViTE: Virtual Graph Trajectory Expert Router for Pedestrian Trajectory Predictioncited by 4
  11. 2024GETS: Ensemble Temperature Scaling for Calibration in Graph Neural Networkscited by 3
  12. 2025MoSE: Unveiling Structural Patterns in Graphs via Mixture of Subgraph Expertscited by 3
  13. 2025Graph Few-Shot Learning via Adaptive Spectrum Experts and Cross-Set Distribution Calibrationcited by 3
  14. 2025Modeling Expert Interactions in Sparse Mixture of Experts via Graph Structurescited by 3
  15. 2025When Structure Doesn't Help: LLMs Do Not Read Text-Attributed Graphs as Effectively as We Expectedcited by 3
  16. 2025Explainable Attention-Guided Stacked Graph Neural Networks for Malware Detectioncited by 2
  17. 2025REN: Anatomically-Informed Mixture-of-Experts for Interstitial Lung Disease Diagnosiscited by 2
  18. 2025Unifying and Enhancing Graph Transformers via a Hierarchical Mask Frameworkcited by 2
  19. 2025GMoPE:A Prompt-Expert Mixture Framework for Graph Foundation Modelscited by 2
  20. 2024Mixture of Scope Experts at Test: Generalizing Deeper Graph Neural Networks with Shallow Variantscited by 1
  21. 2024TopER: Topological Embeddings in Graph Representation Learningcited by 1
  22. 2025AdaptGOT: A Pre-trained Model for Adaptive Contextual POI Representation Learningcited by 1
  23. 2025Leveraging Manifold Embeddings for Enhanced Graph Transformer Representations and Learningcited by 1
  24. 2025Glance for Context: Learning When to Leverage LLMs for Node-Aware GNN-LLM Fusioncited by 1
  25. 2026Topology-Aware Multiscale Mixture of Experts for Efficient Molecular Property Predictioncited by 1
  26. 2026Modality as Heterogeneity: Node Splitting and Graph Rewiring for Multimodal Graph Learningcited by 1
  27. 2026Unified Multi-Domain Graph Pre-training for Homogeneous and Heterogeneous Graphs via Domain-Specific Expert Encodingcited by 1
  28. 2026MxGPS: Multiplex Graph Transformers for a Power Grid Foundation Modelcited by 1
  29. 2026TopoFormer: Topology Meets Attention for Graph Learningcited by 1
  30. 2024M2oE: Multimodal Collaborative Expert Peptide Modelcited by 0
  31. 2025Adaptive Substructure-Aware Expert Model for Molecular Property Predictioncited by 0
  32. 2025HAECcity: Open-Vocabulary Scene Understanding of City-Scale Point Clouds with Superpoint Graph Clusteringcited by 0
  33. 2025Directed Semi-Simplicial Learning with Applications to Brain Activity Decodingcited by 0
  34. 2025Directed Homophily-Aware Graph Neural Networkcited by 0
  35. 2025Out-of-Distribution Graph Models Mergingcited by 0
  36. 2025H2GFM: Towards unifying Homogeneity and Heterogeneity on Text-Attributed Graphscited by 0
  37. 2025Dynamic Adaptive Parsing of Temporal and Cross-Variable Patterns for Network State Classificationcited by 0
  38. 2025Backdoor or Manipulation? Graph Mixture of Experts Can Defend Against Various Graph Adversarial Attackscited by 0

50 most cited of 63 All 63 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Mixture of Experts for Node Classification
  2. 2025Mixture of Raytraced Experts
  3. 2025Dynamic Mixture-of-Experts for Incremental Graph Learning
  4. 2026∞-MoE: Generalizing Mixture of Experts to Infinite Experts
  5. 2024Graph Knowledge Distillation to Mixture of Experts
  6. 2024Mixture of Experts Meets Decoupled Message Passing: Towards General and Adaptive Node Classification
  7. 2026Generalizing GNNs with Tokenized Mixture of Experts
  8. 2026Generalization and Scaling Laws for Mixture-of-Experts Transformers