mlsanityData to 14 Oct 2026

Enhanced Soups for Graph Neural Networks

arXiv 2503.11612 ↗First posted 14 Mar 2025cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Graph Neural Networks (GNN) have demonstrated state-of-the-art performance in numerous scientific and high-performance computing (HPC) applications. Recent work suggests that "souping" (combining) individually trained GNNs into a single model can improve performance without increasing compute and memory costs during inference. However, existing souping algorithms are often slow and memory-intensive, which limits their scalability. We introduce Learned Souping for GNNs, a gradient-descent-based souping strategy that substantially reduces time and memory overhead compared to existing methods. Our approach is evaluated across multiple Open Graph Benchmark (OGB) datasets and GNN architectures, achieving up to 1.2% accuracy improvement and 2.1X speedup. Additionally, we propose Partition Learned Souping, a novel partition-based variant of learned souping that significantly reduces memory usage. On the ogbn-products dataset with GraphSAGE, partition learned souping achieves a 24.5X speedup and a 76% memory reduction without compromising accuracy.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 31 of its 49 references that are in the map. First posted on arXiv 14 Mar 2025.

Builds on

31 of 49 references.

References · 31

  1. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  2. 2017Graph Attention Networkscited by 8,923
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2017Inductive Representation Learning on Large Graphscited by 6,320
  5. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  6. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,163
  7. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  8. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  9. 2021OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphscited by 329
  10. 2023Personalized Soups: Personalized Large Language Model Alignment via Post-hoc Parameter Mergingcited by 280
  11. 2022GraphCast: Learning skillful medium-range global weather forecastingcited by 267
  12. 2021Large-Scale Representation Learning on Graphs via Bootstrappingcited by 186
19 more
  1. 2019Layer-Dependent Importance Sampling for Training Deep and Large Graph Convolutional Networkscited by 177
  2. 2021Training Graph Neural Networks with 1000 Layerscited by 169
  3. 2021Dirichlet Energy Constrained Learning for Deep Graph Neural Networkscited by 97
  4. 2021Accelerating Training and Inference of Graph Neural Networks with Fast Sampling and Pipeliningcited by 39
  5. 2021Learn Locally, Correct Globally: A Distributed Algorithm for Training Graph Neural Networkscited by 24
  6. 2023Seasoning Model Soups for Robustness to Adversarial and Natural Distribution Shiftscited by 23
  7. 2023Instant Soup: Cheap Pruning Ensembles in A Single Pass Can Draw Lottery Tickets from Large Modelscited by 20
  8. 2023Sparse Model Soups: A Recipe for Improved Pruning via Model Averagingcited by 20
  9. 2022Layer-Neighbor Sampling -- Defusing Neighborhood Explosion in GNNscited by 19
  10. 2023FedSoup: Improving Generalization and Personalization in Federated Learning via Selective Model Interpolationcited by 14
  11. 2022Old can be Gold: Better Gradient Flow can Make Vanilla-GCNs Great Againcited by 11
  12. 2023Descriptor and Word Soups: Overcoming the Parameter Efficiency Accuracy Tradeoff for Out-of-Distribution Few-shot Learningcited by 6
  13. 2021On Graph Neural Network Ensembles for Large-Scale Molecular Property Predictioncited by 5
  14. 2023GNN-Ensemble: Towards Random Decision Graph Neural Networkscited by 4
  15. 2024MassiveGNN: Efficient Training via Prefetching for Massively Connected Distributed Graphscited by 3
  16. 2023Graph Ladling: Shockingly Simple Parallel GNN Training without Intermediate Communicationcited by 2
  17. 2023BatchGNN: Efficient CPU-Based Distributed GNN Training on Very Large Graphscited by 2
  18. 2023Do the Frankenstein, or how to achieve better out-of-distribution performance with manifold mixing model soupcited by 1
  19. 2024RADIN: Souping on a Budgetcited by 1

Built on by

Not cited yet.