mlsanityData to 7 Oct 2026

Armada: Memory-Efficient Distributed Training of Large-Scale Graph Neural Networks

arXiv 2502.17846 ↗First posted 25 Feb 2025cs.LG cs.DCcited by 0

Not on the map. Why?

Abstract (arXiv)

We study distributed training of Graph Neural Networks (GNNs) on billion-scale graphs that are partitioned across machines. Efficient training in this setting relies on min-edge-cut partitioning algorithms, which minimize cross-machine communication due to GNN neighborhood sampling. Yet, min-edge-cut partitioning over large graphs remains a challenge: State-of-the-art (SoTA) offline methods (e.g., METIS) are effective, but they require orders of magnitude more memory and runtime than GNN training itself, while computationally efficient algorithms (e.g., streaming greedy approaches) suffer from increased edge cuts. Thus, in this work we introduce Armada, a new end-to-end system for distributed GNN training whose key contribution is GREM, a novel min-edge-cut partitioning algorithm that can efficiently scale to large graphs. GREM builds on streaming greedy approaches with one key addition: prior vertex assignments are continuously refined during streaming, rather than frozen after an initial greedy selection. Our theoretical analysis and experimental results show that this refinement is critical to minimizing edge cuts and enables GREM to reach partition quality comparable to METIS but with 8-65x less memory and 8-46x faster. Given a partitioned graph, Armada leverages a new disaggregated architecture for distributed GNN training to further improve efficiency; we find that on common cloud machines, even with zero communication, GNN neighborhood sampling and feature loading bottleneck training. Disaggregation allows Armada to independently allocate resources for these operations and ensure that expensive GPUs remain saturated with computation. We evaluate Armada against SoTA systems for distributed GNN training and find that the disaggregated architecture leads to runtime improvements up to 4.5x and cost reductions up to 3.1x.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 31 of its 55 references that are in the map. First posted on arXiv 25 Feb 2025.

Builds on

31 of 55 references.

References · 31

  1. 2017Inductive Representation Learning on Large Graphscited by 6,284
  2. 2019A Comprehensive Survey on Graph Neural Networkscited by 2,906
  3. 2019Fast Graph Representation Learning with PyTorch Geometriccited by 2,414
  4. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,148
  5. 2019GraphSAINT: Graph Sampling Based Inductive Learning Methodcited by 618
  6. 2018FastGCN: Fast Learning with Graph Convolutional Networks via Importance Samplingcited by 583
  7. 2021OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphscited by 328
  8. 2022GraphCast: Learning skillful medium-range global weather forecastingcited by 261
  9. 2019Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networkscited by 241
  10. 2020Machine Learning on Graphs: A Model and Comprehensive Taxonomycited by 156
  11. 2019AliGraph: A Comprehensive Graph Neural Network Platformcited by 152
  12. 2020DistDGL: Distributed Graph Neural Network Training for Billion-Scale Graphscited by 139
19 more
  1. 2021ETA Prediction with Graph Neural Networks in Google Mapscited by 130
  2. 2019PyTorch-BigGraph: A Large-scale Graph Embedding Systemcited by 118
  3. 2020DGL-KE: Training Knowledge Graph Embeddings at Scalecited by 67
  4. 2021BGL: GPU-Efficient GNN Training by Optimizing Graph Data I/O and Preprocessingcited by 50
  5. 2021Accelerating Training and Inference of Graph Neural Networks with Fast Sampling and Pipeliningcited by 39
  6. 2021Distributed Hybrid CPU and GPU training for Graph Neural Networks on Billion-Scale Graphscited by 33
  7. 2022Distributed Graph Neural Network Training: A Surveycited by 32
  8. 2020Accelerating Graph Sampling for Graph Machine Learning using GPUscited by 29
  9. 2021Marius: Learning Massive Graph Embeddings on a Single Machinecited by 29
  10. 2021Large Graph Convolutional Network Training with GPU-Oriented Data Communication Architecturecited by 25
  11. 2021Global Neighbor Sampling for Mixed CPU-GPU Training on Giant Graphscited by 22
  12. 2022MariusGNN: Resource-Efficient Out-of-Core Training of Graph Neural Networkscited by 19
  13. 2023Communication-Efficient Graph Neural Networks with Probabilistic Neighborhood Expansion Analysis and Cachingcited by 16
  14. 2022A Comprehensive Survey on Distributed Training of Graph Neural Networkscited by 15
  15. 2021PyTorch-Direct: Enabling GPU Centric Data Access for Very Large Graph Neural Network Training with Irregular Accessescited by 12
  16. 2019Window-based Streaming Graph Partitioning Algorithmcited by 4
  17. 2023An Experimental Comparison of Partitioning Strategies for Distributed Graph Neural Network Trainingcited by 4
  18. 2021Buffered Streaming Graph Partitioningcited by 2
  19. 2023Communication-Free Distributed GNN Training with Vertex Cutcited by 2

Built on by

Not cited yet.