MassiveGNN: Efficient Training via Prefetching for Massively Connected Distributed Graphs
Not on the map. Why?
Abstract (arXiv)
Graph Neural Networks (GNN) are indispensable in learning from graph-structured data, yet their rising computational costs, especially on massively connected graphs, pose significant challenges in terms of execution performance. To tackle this, distributed-memory solutions such as partitioning the graph to concurrently train multiple replicas of GNNs are in practice. However, approaches requiring a partitioned graph usually suffer from communication overhead and load imbalance, even under optimal partitioning and communication strategies due to irregularities in the neighborhood minibatch sampling. This paper proposes practical trade-offs for improving the sampling and communication overheads for representation learning on distributed graphs (using popular GraphSAGE architecture) by developing a parameterized continuous prefetch and eviction scheme on top of the state-of-the-art Amazon DistDGL distributed GNN framework, demonstrating about 15-40% improvement in end-to-end training performance on the National Energy Research Scientific Computing Center's (NERSC) Perlmutter supercomputer for various OGB datasets.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 18 of its 30 references that are in the map. First posted on arXiv 30 Oct 2024.
Builds on
18 of 30 references.
References · 18
- 2017Inductive Representation Learning on Large Graphscited by 6,284
- 2019A Comprehensive Survey on Graph Neural Networkscited by 2,906
- 2019Fast Graph Representation Learning with PyTorch Geometriccited by 2,414
- 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,148
- 2018Graph Neural Networks: A Review of Methods and Applicationscited by 1,571
- 2019GraphSAINT: Graph Sampling Based Inductive Learning Methodcited by 618
- 2021Graph Neural Network for Traffic Forecasting: A Surveycited by 260
- 2019AliGraph: A Comprehensive Graph Neural Network Platformcited by 152
- 2020DistDGL: Distributed Graph Neural Network Training for Billion-Scale Graphscited by 139
- 2021BGL: GPU-Efficient GNN Training by Optimizing Graph Data I/O and Preprocessingcited by 50
- 2022Distributed Graph Neural Network Training: A Surveycited by 32
- 2013Recent Advances in Graph Partitioningcited by 24
6 more
- 2021Learn Locally, Correct Globally: A Distributed Algorithm for Training Graph Neural Networkscited by 24
- 2022MariusGNN: Resource-Efficient Out-of-Core Training of Graph Neural Networkscited by 19
- 2022Layer-Neighbor Sampling -- Defusing Neighborhood Explosion in GNNscited by 19
- 2023Communication-Efficient Graph Neural Networks with Probabilistic Neighborhood Expansion Analysis and Cachingcited by 16
- 2023Distributed Matrix-Based Sampling for Graph Neural Network Trainingcited by 6
- 2023BatchGNN: Efficient CPU-Based Distributed GNN Training on Very Large Graphscited by 2
Built on by
Citing papers per year: 2025: 1; 2026: 2.
3 papers.