mlsanityData to 7 Oct 2026

GNNPipe: Scaling Deep GNN Training with Pipelined Model Parallelism

arXiv 2308.10087 ↗First posted 19 Aug 2023cs.DC cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Communication is a key bottleneck for distributed graph neural network (GNN) training. This paper proposes GNNPipe, a new approach that scales the distributed full-graph deep GNN training. Being the first to use layer-level model parallelism for GNN training, GNNPipe partitions GNN layers among GPUs, each device performs the computation for a disjoint subset of consecutive GNN layers on the whole graph. Compared to graph parallelism with each GPU handling a graph partition, GNNPipe reduces the communication volume by a factor of the number of GNN layers. GNNPipe overcomes the unique challenges for pipelined layer-level model parallelism on the whole graph by partitioning it into dependent chunks, allowing the use of historical vertex embeddings, and applying specific training techniques to ensure convergence. We also propose a hybrid approach by combining GNNPipe with graph parallelism to handle large graphs, achieve better computer resource utilization and ensure model convergence. We build a general GNN training system supporting all three parallelism setting. Extensive experiments show that our method reduces the per-epoch training time by up to 2.45x (on average 1.58x) and reduces the communication volume and overhead by up to 22.89x and 27.21x (on average 8.69x and 11.60x), respectively, while achieving a comparable level of model accuracy and convergence speed compared to graph parallelism.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 46 of its 70 references that are in the map. First posted on arXiv 19 Aug 2023.

Builds on

46 of 70 references.

References · 46

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  5. 2017Graph Attention Networkscited by 8,854
  6. 2016Densely Connected Convolutional Networkscited by 6,691
  7. 2017Inductive Representation Learning on Large Graphscited by 6,284
  8. 2016Layer Normalizationcited by 5,579
  9. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  10. 2018How Powerful are Graph Neural Networks?cited by 4,355
  11. 2014Going Deeper with Convolutionscited by 4,231
  12. 2017Neural Message Passing for Quantum Chemistrycited by 3,460
34 more
  1. 2019Fast Graph Representation Learning with PyTorch Geometriccited by 2,414
  2. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,220
  3. 2018Pitfalls of Graph Neural Network Evaluationcited by 994
  4. 2018Representation Learning on Graphs with Jumping Knowledge Networkscited by 875
  5. 2020Simple and Deep Graph Convolutional Networkscited by 857
  6. 2019DropEdge: Towards Deep Graph Convolutional Networks on Node Classificationcited by 679
  7. 2019GraphSAINT: Graph Sampling Based Inductive Learning Methodcited by 618
  8. 2018FastGCN: Fast Learning with Graph Convolutional Networks via Importance Samplingcited by 583
  9. 2019Multi-scale Attributed Node Embeddingcited by 543
  10. 2019Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networkscited by 540
  11. 2019DeepGCNs: Can GCNs Go as Deep as CNNs?cited by 527
  12. 2014One weird trick for parallelizing convolutional neural networkscited by 264
  13. 2020DeeperGCN: All You Need to Train Deeper GCNscited by 249
  14. 2019Layer-Dependent Importance Sampling for Training Deep and Large Graph Convolutional Networkscited by 177
  15. 2021Training Graph Neural Networks with 1000 Layerscited by 169
  16. 2020DistDGL: Distributed Graph Neural Network Training for Billion-Scale Graphscited by 139
  17. 2020Memory-Efficient Pipeline-Parallel DNN Trainingcited by 133
  18. 2020Towards Deeper Graph Neural Networks with Differentiable Group Normalizationcited by 113
  19. 2020DAPPLE: A Pipelined Data Parallel Approach for Training Large Modelscited by 100
  20. 2022Decoupling the Depth and Scope of Graph Neural Networkscited by 94
  21. 2021GNNAutoScale: Scalable and Expressive Graph Neural Networks via Historical Embeddingscited by 90
  22. 2020Bayesian Graph Neural Networks with Adaptive Connection Samplingcited by 81
  23. 2019DeepGCNs: Making GCNs Go as Deep as CNNscited by 72
  24. 2021Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelinescited by 68
  25. 2021Dorylus: Affordable, Scalable, and Accurate GNN Training with Distributed CPU Servers and Serverless Threadscited by 51
  26. 2021BGL: GPU-Efficient GNN Training by Optimizing Graph Data I/O and Preprocessingcited by 50
  27. 2020Neural Subgraph Matchingcited by 47
  28. 2022PipeGCN: Efficient Full-Graph Training of Graph Convolutional Networks with Pipelined Feature Communicationcited by 45
  29. 2022BNS-GCN: Efficient Full-Graph Training of Graph Convolutional Networks with Partition-Parallelism and Random Boundary Node Samplingcited by 43
  30. 2022Distributed Graph Neural Network Training: A Surveycited by 32
  31. 2021Sequential Aggregation and Rematerialization: Distributed Full-batch Training of Graph Neural Networks on Large Graphscited by 18
  32. 2022MGG: Accelerating Graph Neural Networks with Fine-grained intra-kernel Communication-Computation Pipelining on Multi-GPU Platformscited by 16
  33. 2023Adaptive Message Quantization and Parallelization for Distributed Full-graph GNN Trainingcited by 10
  34. 2023GSplit: Scaling Graph Neural Network Training on Large Graphs via Split-Parallelismcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 2.

4 papers.

  1. 2024Graph Neural Network Training Systems: A Performance Comparison of Full-Graph and Mini-Batchcited by 10
  2. 2025Forecasting at Full Spectrum: Holistic Multi-Granular Traffic Modeling under High-Throughput Inference Regimescited by 1
  3. 2026MQ-GNN: A Multi-Queue Pipelined Architecture for Scalable and Efficient GNN Trainingcited by 0
  4. 2026Communication-free Sampling and 4D Hybrid Parallelism for Scalable Mini-batch GNN Trainingcited by 0