# GraphMETRO: Mitigating Complex Graph Distribution Shifts via Mixture of Aligned Experts

arXiv: https://arxiv.org/abs/2312.04693 · HTML: https://mlsanity.com/paper/2312.04693/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in MoE for time series and graphs, one of 12 directions in Mixture-of-experts.
- Builds on 66 of its 87 references that are in the map.
- First posted on arXiv 7 Dec 2023.

## Placed in

- Mixture-of-experts / [MoE for time series and graphs](https://mlsanity.com/moe/moe-for-time-series-and-graphs/)

## Builds on (most cited first, 30 of 66)

- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalization](https://mlsanity.com/paper/1911.08731/) (2019)
- [Deep CORAL: Correlation Alignment for Deep Domain Adaptation](https://mlsanity.com/paper/1607.01719/) (2016)
- [Graph Convolutional Neural Networks for Web-Scale Recommender Systems](https://mlsanity.com/paper/1806.01973/) (2018)
- [Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss](https://mlsanity.com/paper/1906.07413/) (2019)
- [Revisiting Semi-Supervised Learning with Graph Embeddings](https://mlsanity.com/paper/1603.08861/) (2016)
- [GLaM: Efficient Scaling of Language Models with Mixture-of-Experts](https://mlsanity.com/paper/2112.06905/) (2021)
- [Geom-GCN: Geometric Graph Convolutional Networks](https://mlsanity.com/paper/2002.05287/) (2020)
- [Scaling Vision with Sparse Mixture of Experts](https://mlsanity.com/paper/2106.05974/) (2021)
- [Out-of-Distribution Generalization via Risk Extrapolation (REx)](https://mlsanity.com/paper/2003.00688/) (2020)
- [TUDataset: A collection of benchmark datasets for learning with graphs](https://mlsanity.com/paper/2007.08663/) (2020)
- [DropEdge: Towards Deep Graph Convolutional Networks on Node Classification](https://mlsanity.com/paper/1907.10903/) (2019)
- [Benchmarking Graph Neural Networks](https://mlsanity.com/paper/2003.00982/) (2020)
- [Long-tail learning via logit adjustment](https://mlsanity.com/paper/2007.07314/) (2020)
- [Explainability in Graph Neural Networks: A Taxonomic Survey](https://mlsanity.com/paper/2012.15445/) (2020)
- [MAGNN: Metapath Aggregated Graph Neural Network for Heterogeneous Graph Embedding](https://mlsanity.com/paper/2002.01680/) (2020)
- [Graph Random Neural Network for Semi-Supervised Learning on Graphs](https://mlsanity.com/paper/2005.11079/) (2020)
- [Data Augmentation for Graph Neural Networks](https://mlsanity.com/paper/2006.06830/) (2020)
- [Discovering Invariant Rationales for Graph Neural Networks](https://mlsanity.com/paper/2201.12872/) (2022)
- [Interpretable and Generalizable Graph Learning via Stochastic Attention Mechanism](https://mlsanity.com/paper/2201.12987/) (2022)
- [Understanding Attention and Generalization in Graph Neural Networks](https://mlsanity.com/paper/1905.02850/) (2019)

## Built on by (most cited first, 19 of 19)

- [A Survey of Large Language Models for Text-Guided Molecular Discovery: from Molecule Generation to Optimization](https://mlsanity.com/paper/2505.16094/) (2025)
- [GRAVER: Generative Graph Vocabularies for Robust Graph Foundation Models Fine-tuning](https://mlsanity.com/paper/2511.05592/) (2025)
- [Safety in Graph Machine Learning: Threats and Safeguards](https://mlsanity.com/paper/2405.11034/) (2024)
- [Revisiting, Benchmarking and Understanding Unsupervised Graph Domain Adaptation](https://mlsanity.com/paper/2407.11052/) (2024)
- [Hierarchical Mixture of Experts: Generalizable Learning for High-Level Synthesis](https://mlsanity.com/paper/2410.19225/) (2024)
- [DA-MoE: Addressing Depth-Sensitivity in Graph-Level Analysis through Mixture of Experts](https://mlsanity.com/paper/2411.03025/) (2024)
- [Efficient Task Transfer for HLS DSE](https://mlsanity.com/paper/2408.13270/) (2024)
- [Graph Few-Shot Learning via Adaptive Spectrum Experts and Cross-Set Distribution Calibration](https://mlsanity.com/paper/2510.12140/) (2025)
- [Can Molecular Foundation Models Know What They Don't Know? A Simple Remedy with Preference Optimization](https://mlsanity.com/paper/2509.25509/) (2025)
- [Leveraging Manifold Embeddings for Enhanced Graph Transformer Representations and Learning](https://mlsanity.com/paper/2507.07335/) (2025)
- [Context-aware Graph Causality Inference for Few-Shot Molecular Property Prediction](https://mlsanity.com/paper/2601.11135/) (2026)
- [Topology-Aware Multiscale Mixture of Experts for Efficient Molecular Property Prediction](https://mlsanity.com/paper/2601.12637/) (2026)
- [Generalizing GNNs with Tokenized Mixture of Experts](https://mlsanity.com/paper/2602.09258/) (2026)
- [Backdoor or Manipulation? Graph Mixture of Experts Can Defend Against Various Graph Adversarial Attacks](https://mlsanity.com/paper/2510.15333/) (2025)
- [Training Diverse Graph Experts for Ensembles: A Systematic Empirical Study](https://mlsanity.com/paper/2510.18370/) (2025)
- [IDOL: Meeting Diverse Distribution Shifts with Prior Physics for Tropical Cyclone Multi-Task Estimation](https://mlsanity.com/paper/2511.11750/) (2025)
- [Self-Adaptive Graph Mixture of Models](https://mlsanity.com/paper/2511.13062/) (2025)
- [MalMoE: Mixture-of-Experts Enhanced Encrypted Malicious Traffic Detection Under Graph Drift](https://mlsanity.com/paper/2602.10157/) (2026)
- [MoEMambaMIL: Structure-Aware Selective State Space Modeling for Whole-Slide Image Analysis](https://mlsanity.com/paper/2603.06378/) (2026)

## Abstract (arXiv)

Graph data are inherently complex and heterogeneous, leading to a high natural diversity of distributional shifts. However, it remains unclear how to build machine learning architectures that generalize to the complex distributional shifts naturally occurring in the real world. Here, we develop GraphMETRO, a Graph Neural Network architecture that models natural diversity and captures complex distributional shifts. GraphMETRO employs a Mixture-of-Experts (MoE) architecture with a gating model and multiple expert models, where each expert model targets a specific distributional shift to produce a referential representation w.r.t. a reference model, and the gating model identifies shift components. Additionally, we design a novel objective that aligns the representations from different expert models to ensure reliable optimization. GraphMETRO achieves state-of-the-art results on four datasets from the GOOD benchmark, which is comprised of complex and natural real-world distribution shifts, improving by 67% and 4.2% on the WebKB and Twitch datasets. Code and data are available at https://github.com/Wuyxin/GraphMETRO.