mlsanityData to 7 Oct 2026

A Comprehensive Survey on Distributed Training of Graph Neural Networks

arXiv 2211.05368 ↗First posted 10 Nov 2022cs.DC cs.LGcited by 15

Not on the map. Why?

Abstract (arXiv)

Graph neural networks (GNNs) have been demonstrated to be a powerful algorithmic model in broad application fields for their effectiveness in learning over graphs. To scale GNN training up for large-scale and ever-growing graphs, the most promising solution is distributed training which distributes the workload of training across multiple computing nodes. At present, the volume of related research on distributed GNN training is exceptionally vast, accompanied by an extraordinarily rapid pace of publication. Moreover, the approaches reported in these studies exhibit significant divergence. This situation poses a considerable challenge for newcomers, hindering their ability to grasp a comprehensive understanding of the workflows, computational patterns, communication strategies, and optimization techniques employed in distributed GNN training. As a result, there is a pressing need for a survey to provide correct recognition, analysis, and comparisons in this field. In this paper, we provide a comprehensive survey of distributed GNN training by investigating various optimization techniques used in distributed GNN training. First, distributed GNN training is classified into several categories according to their workflows. In addition, their computational patterns and communication patterns, as well as the optimization techniques proposed by recent work are introduced. Second, the software frameworks and hardware platforms of distributed GNN training are also introduced for a deeper understanding. Third, distributed GNN training is compared with distributed training of deep neural networks, emphasizing the uniqueness of distributed GNN training. Finally, interesting issues and opportunities in this field are discussed.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 74 of its 159 references that are in the map. First posted on arXiv 10 Nov 2022.

Builds on

74 of 159 references.

References · 74

  1. 2017Attention Is All You Needcited by 59,932
  2. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  3. 2017Graph Attention Networkscited by 8,854
  4. 2017Inductive Representation Learning on Large Graphscited by 6,284
  5. 2018How Powerful are Graph Neural Networks?cited by 4,355
  6. 2017Neural Message Passing for Quantum Chemistrycited by 3,460
  7. 2019A Comprehensive Survey on Graph Neural Networkscited by 2,906
  8. 2019Fast Graph Representation Learning with PyTorch Geometriccited by 2,414
  9. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,148
  10. 2017Modeling Relational Data with Graph Convolutional Networkscited by 1,684
  11. 2018Graph Neural Networks: A Review of Methods and Applicationscited by 1,571
  12. 2018Relational inductive biases, deep learning, and graph networkscited by 1,341
38 more
  1. 2018Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognitioncited by 1,199
  2. 2018Graph Convolutional Neural Networks for Web-Scale Recommender Systemscited by 1,133
  3. 2016Geometric deep learning: going beyond Euclidean datacited by 1,042
  4. 2018Predict then Propagate: Graph Neural Networks meet Personalized PageRankcited by 987
  5. 2020Simple and Deep Graph Convolutional Networkscited by 857
  6. 2018Link Prediction Based on Graph Neural Networkscited by 829
  7. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  8. 2019Graph Neural Networks for Social Recommendationcited by 711
  9. 2019DropEdge: Towards Deep Graph Convolutional Networks on Node Classificationcited by 679
  10. 2020On the Bottleneck of Graph Neural Networks and its Practical Implicationscited by 624
  11. 2019GraphSAINT: Graph Sampling Based Inductive Learning Methodcited by 618
  12. 2018FastGCN: Fast Learning with Graph Convolutional Networks via Importance Samplingcited by 583
  13. 2020Connecting the Dots: Multivariate Time Series Forecasting with Graph Neural Networkscited by 563
  14. 2019Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networkscited by 540
  15. 2018Deep Learning on Graphs: A Surveycited by 393
  16. 2017Recent Trends in Deep Learning Based Natural Language Processingcited by 262
  17. 2020DeeperGCN: All You Need to Train Deeper GCNscited by 249
  18. 2019Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networkscited by 241
  19. 2020Spatial-Temporal Fusion Graph Neural Networks for Traffic Flow Forecastingcited by 212
  20. 2018Adaptive Sampling Towards Fast Graph Representation Learningcited by 180
  21. 2019Layer-Dependent Importance Sampling for Training Deep and Large Graph Convolutional Networkscited by 177
  22. 2020Machine Learning on Graphs: A Model and Comprehensive Taxonomycited by 156
  23. 2019AliGraph: A Comprehensive Graph Neural Network Platformcited by 152
  24. 2020DistDGL: Distributed Graph Neural Network Training for Billion-Scale Graphscited by 139
  25. 2021PyTorch Geometric Temporal: Spatiotemporal Signal Processing with Neural Machine Learning Modelscited by 118
  26. 2020Graph Neural Networks Meet Neural-Symbolic Computing: A Survey and Perspectivecited by 93
  27. 2019Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerializationcited by 90
  28. 2020GCN-RL Circuit Designer: Transferable Transistor Sizing with Graph Neural Networks and Reinforcement Learningcited by 89
  29. 2020Computing Graph Neural Networks: A Survey from Algorithms to Acceleratorscited by 81
  30. 2020HyGCN: A GCN Accelerator with Hybrid Architecturecited by 73
  31. 2017Graph Convolutional Encoders for Syntax-aware Neural Machine Translationcited by 70
  32. 2020Tackling Over-Smoothing for General Graph Convolutional Networkscited by 66
  33. 2021DistGNN: Scalable Distributed Training for Large-Scale Graph Neural Networkscited by 59
  34. 2022TGL: A General Framework for Temporal GNN Training on Billion-Scale Graphscited by 59
  35. 2019Spam Review Detection with Graph Convolutional Networkscited by 57
  36. 2021Dorylus: Affordable, Scalable, and Accurate GNN Training with Distributed CPU Servers and Serverless Threadscited by 51
  37. 2020Reducing Communication in Graph Neural Network Trainingcited by 50
  38. 2019Memory Augmented Graph Neural Networks for Sequential Recommendationcited by 46

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 10; 2025: 2; 2026: 2.

15 papers.

Sort:

  1. 2023Comprehensive Evaluation of GNN Training Systems: A Data Management Perspectivecited by 9
  2. 2024A Systematic Literature Review of Spatio-Temporal Graph Neural Network Models for Time Series Forecasting and Classificationcited by 9
  3. 2025Graph Neural Network-Based Collaborative Perception for Adaptive Scheduling in Distributed Systemscited by 4
  4. 2024A Structure-Aware Framework for Learning Device Placements on Computation Graphscited by 3
  5. 2024Fully Distributed Online Training of Graph Neural Networks in Networked Systemscited by 3
  6. 2024Revisiting Edge Perturbation for Graph Neural Network in Graph Data Augmentation and Attackcited by 2
  7. 2024Edge Graph Intelligence: Reciprocally Empowering Edge Networks with Graph Intelligencecited by 2
  8. 2024Model Agnostic Hybrid Sharding For Heterogeneous Distributed Inferencecited by 2
  9. 2024Disttack: Graph Adversarial Attacks Toward Distributed GNN Trainingcited by 1
  10. 2024LSM-GNN: Large-scale Storage-based Multi-GPU GNN Training by Optimizing Data Transfer Schemecited by 1
  11. 2024Analysis of Gene Regulatory Networks from Gene Expression Using Graph Neural Networkscited by 1
  12. 2024SiHGNN: Leveraging Properties of Semantic Graphs for Efficient HGNN Accelerationcited by 0
3 more
  1. 2025Armada: Memory-Efficient Distributed Training of Large-Scale Graph Neural Networkscited by 0
  2. 2026Communication-free Sampling and 4D Hybrid Parallelism for Scalable Mini-batch GNN Trainingcited by 0
  3. 2026SNI-GNN: SmartNIC-Assisted Full-Graph GNN Training with In-Network Embedding Predictioncited by 0