mlsanityData to 14 Oct 2026

GSplit: Scaling Graph Neural Network Training on Large Graphs via Split-Parallelism

arXiv 2303.13775 ↗First posted 24 Mar 2023cs.DC cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Graph neural networks (GNNs), an emerging class of machine learning models for graphs, have gained popularity for their superior performance in various graph analytical tasks. Mini-batch training is commonly used to train GNNs on large graphs, and data parallelism is the standard approach to scale mini-batch training across multiple GPUs. Data parallel approaches contain redundant work as subgraphs sampled by different GPUs contain significant overlap. To address this issue, we introduce a hybrid parallel mini-batch training paradigm called split parallelism. Split parallelism avoids redundant work by splitting the sampling, loading, and training of each mini-batch across multiple GPUs. Split parallelism, however, introduces communication overheads that can be more than the savings from removing redundant work. We further present a lightweight partitioning algorithm that probabilistically minimizes these overheads. We implement split parallelism in GSplit and show that it outperforms state-of-the-art mini-batch training systems like DGL, Quiver, and P3.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 29 of its 80 references that are in the map. First posted on arXiv 24 Mar 2023.

Builds on

29 of 80 references.

References · 29

  1. 2017Graph Attention Networkscited by 8,923
  2. 2017Inductive Representation Learning on Large Graphscited by 6,320
  3. 2017Neural Message Passing for Quantum Chemistrycited by 3,487
  4. 2019Fast Graph Representation Learning with PyTorch Geometriccited by 2,434
  5. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,163
  6. 2018Graph Convolutional Neural Networks for Web-Scale Recommender Systemscited by 1,138
  7. 2019GraphSAINT: Graph Sampling Based Inductive Learning Methodcited by 618
  8. 2021OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphscited by 329
  9. 2020SIGN: Scalable Inception Graph Neural Networkscited by 284
  10. 2019Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networkscited by 242
  11. 2019AliGraph: A Comprehensive Graph Neural Network Platformcited by 153
  12. 2020DistDGL: Distributed Graph Neural Network Training for Billion-Scale Graphscited by 139
17 more
  1. 2021DistGNN: Scalable Distributed Training for Large-Scale Graph Neural Networkscited by 59
  2. 2021Dorylus: Affordable, Scalable, and Accurate GNN Training with Distributed CPU Servers and Serverless Threadscited by 51
  3. 2021BGL: GPU-Efficient GNN Training by Optimizing Graph Data I/O and Preprocessingcited by 50
  4. 2022SparseTIR: Composable Abstractions for Sparse Compilation in Deep Learningcited by 47
  5. 2020An Introduction to Electrocatalyst Design using Machine Learning for Renewable Energy Storagecited by 45
  6. 2022PipeGCN: Efficient Full-Graph Training of Graph Convolutional Networks with Pipelined Feature Communicationcited by 45
  7. 2022BNS-GCN: Efficient Full-Graph Training of Graph Convolutional Networks with Partition-Parallelism and Random Boundary Node Samplingcited by 43
  8. 2021Accelerating Training and Inference of Graph Neural Networks with Fast Sampling and Pipeliningcited by 39
  9. 2022Distributed Graph Neural Network Training: A Surveycited by 32
  10. 2020Accelerating Graph Sampling for Graph Machine Learning using GPUscited by 29
  11. 2020Utilising Graph Machine Learning within Drug Discovery and Developmentcited by 26
  12. 2021Large Graph Convolutional Network Training with GPU-Oriented Data Communication Architecturecited by 25
  13. 2021Global Neighbor Sampling for Mixed CPU-GPU Training on Giant Graphscited by 22
  14. 2021TC-GNN: Bridging Sparse GNN Computation and Dense Tensor Cores on GPUscited by 17
  15. 2022MGG: Accelerating Graph Neural Networks with Fine-grained intra-kernel Communication-Computation Pipelining on Multi-GPU Platformscited by 16
  16. 2020C-SAW: A Framework for Graph Sampling and Random Walk on GPUscited by 13
  17. 2023Boosting Distributed Full-graph GNN Training with Asynchronous One-bit Communicationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 2; 2025: 1; 2026: 1.

6 papers.

  1. 2024Graph Neural Network Training Systems: A Performance Comparison of Full-Graph and Mini-Batchcited by 10
  2. 2024Two Trades is not Baffled: Condensing Graph via Crafting Rational Gradient Matchingcited by 8
  3. 2023GNNPipe: Scaling Deep GNN Training with Pipelined Model Parallelismcited by 4
  4. 2025Deal: Distributed End-to-End GNN Inference for All Nodescited by 1
  5. 2023Cooperative Minibatching in Graph Neural Networkscited by 0
  6. 2026Communication-free Sampling and 4D Hybrid Parallelism for Scalable Mini-batch GNN Trainingcited by 0