# Out-of-Distribution Graph Models Merging

arXiv: https://arxiv.org/abs/2506.03674 · HTML: https://mlsanity.com/paper/2506.03674/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 58 of its 69 references that are in the map.
- First posted on arXiv 4 Jun 2025.

## Builds on (most cited first, 30 of 58)

- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Open Graph Benchmark: Datasets for Machine Learning on Graphs](https://mlsanity.com/paper/2005.00687/) (2020)
- [Invariant Risk Minimization](https://mlsanity.com/paper/1907.02893/) (2019)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [Editing Models with Task Arithmetic](https://mlsanity.com/paper/2212.04089/) (2022)
- [Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfer](https://mlsanity.com/paper/1612.03928/) (2016)
- [Adafactor: Adaptive Learning Rates with Sublinear Memory Cost](https://mlsanity.com/paper/1804.04235/) (2018)
- [Merging Models with Fisher-Weighted Averaging](https://mlsanity.com/paper/2111.09832/) (2021)
- [Multi-scale Attributed Node Embedding](https://mlsanity.com/paper/1909.13021/) (2019)
- [Dreaming to Distill: Data-free Knowledge Transfer via DeepInversion](https://mlsanity.com/paper/1912.08795/) (2019)
- [AdaMerging: Adaptive Model Merging for Multi-Task Learning](https://mlsanity.com/paper/2310.02575/) (2023)
- [How Neural Networks Extrapolate: From Feedforward to Graph Neural Networks](https://mlsanity.com/paper/2009.11848/) (2020)
- [Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalization](https://mlsanity.com/paper/2106.06607/) (2021)
- [ZipIt! Merging Models from Different Tasks without Training](https://mlsanity.com/paper/2305.03053/) (2023)
- [Graph Condensation for Graph Neural Networks](https://mlsanity.com/paper/2110.07580/) (2021)
- [Invariant Rationalization](https://mlsanity.com/paper/2003.09772/) (2020)
- [Dynamic Model Pruning with Feedback](https://mlsanity.com/paper/2006.07253/) (2020)
- [Fusing finetuned models for better pretraining](https://mlsanity.com/paper/2204.03044/) (2022)
- [Causal Attention for Interpretable and Generalizable Graph Classification](https://mlsanity.com/paper/2112.15089/) (2021)
- [Neural Networks Fail to Learn Periodic Functions and How to Fix It](https://mlsanity.com/paper/2006.08195/) (2020)
- [Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks](https://mlsanity.com/paper/2010.02066/) (2020)
- [Out-Of-Distribution Generalization on Graphs: A Survey](https://mlsanity.com/paper/2202.07987/) (2022)
- [Towards a Theoretical Framework of Out-of-Distribution Generalization](https://mlsanity.com/paper/2106.04496/) (2021)
- [Masking as an Efficient Alternative to Finetuning for Pretrained Language Models](https://mlsanity.com/paper/2004.12406/) (2020)
- [Mixture of A Million Experts](https://mlsanity.com/paper/2407.04153/) (2024)
- [Generalizing Graph Neural Networks on Out-Of-Distribution Graphs](https://mlsanity.com/paper/2111.10657/) (2021)
- [A Survey on Deep Graph Generation: Methods and Applications](https://mlsanity.com/paper/2203.06714/) (2022)

## Abstract (arXiv)

This paper studies a novel problem of out-of-distribution graph models merging, which aims to construct a generalized model from multiple graph models pre-trained on different domains with distribution discrepancy. This problem is challenging because of the difficulty in learning domain-invariant knowledge implicitly in model parameters and consolidating expertise from potentially heterogeneous GNN backbones. In this work, we propose a graph generation strategy that instantiates the mixture distribution of multiple domains. Then, we merge and fine-tune the pre-trained graph models via a MoE module and a masking mechanism for generalized adaptation. Our framework is architecture-agnostic and can operate without any source/target domain data. Both theoretical analysis and experimental results demonstrate the effectiveness of our approach in addressing the model generalization problem.