mlsanityData to 7 Oct 2026

Learn Locally, Correct Globally: A Distributed Algorithm for Training Graph Neural Networks

arXiv 2111.08202 ↗First posted 16 Nov 2021cs.LGcited by 24

Not on the map. Why?

Abstract (arXiv)

Despite the recent success of Graph Neural Networks (GNNs), training GNNs on large graphs remains challenging. The limited resource capacities of the existing servers, the dependency between nodes in a graph, and the privacy concern due to the centralized storage and model learning have spurred the need to design an effective distributed algorithm for GNN training. However, existing distributed GNN training methods impose either excessive communication costs or large memory overheads that hinders their scalability. To overcome these issues, we propose a communication-efficient distributed GNN training technique named textLearn Locally, Correct Globally (LLCG). To reduce the communication and memory overhead, each local machine in LLCG first trains a GNN on its local data by ignoring the dependency between nodes among different machines, then sends the locally trained model to the server for periodic model averaging. However, ignoring node dependency could result in significant performance degradation. To solve the performance degradation, we propose to apply textGlobal Server Corrections on the server to refine the locally learned models. We rigorously analyze the convergence of distributed methods with periodic model averaging for training GNNs and show that naively applying periodic model averaging but ignoring the dependency between nodes will suffer from an irreducible residual error. However, this residual error can be eliminated by utilizing the proposed global corrections to entail fast convergence rate. Extensive experiments on real-world datasets show that LLCG can significantly improve the efficiency without hurting the performance.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 30 of its 43 references that are in the map. First posted on arXiv 16 Nov 2021.

Builds on

30 of 43 references.

References · 30

  1. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  2. 2017Graph Attention Networkscited by 8,854
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2017Inductive Representation Learning on Large Graphscited by 6,284
  5. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,148
  6. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
  7. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  8. 2018Graph Convolutional Neural Networks for Web-Scale Recommender Systemscited by 1,133
  9. 2018Predict then Propagate: Graph Neural Networks meet Personalized PageRankcited by 987
  10. 2019Towards Federated Learning at Scale: System Designcited by 826
  11. 2019GraphSAINT: Graph Sampling Based Inductive Learning Methodcited by 618
  12. 2018Local SGD Converges Fast and Communicates Littlecited by 595
18 more
  1. 2018Federated Learning for Mobile Keyboard Predictioncited by 565
  2. 2019Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networkscited by 540
  3. 2019DeepGCNs: Can GCNs Go as Deep as CNNs?cited by 527
  4. 2021OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphscited by 328
  5. 2018Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learningcited by 277
  6. 2019Layer-Dependent Importance Sampling for Training Deep and Large Graph Convolutional Networkscited by 177
  7. 2020DistDGL: Distributed Graph Neural Network Training for Billion-Scale Graphscited by 139
  8. 2018Billion-scale Commodity Embedding for E-commerce Recommendation in Alibabacited by 105
  9. 2021DistGNN: Scalable Distributed Training for Large-Scale Graph Neural Networkscited by 59
  10. 2018Graph Transformation Policy Network for Chemical Reaction Predictioncited by 56
  11. 2020Minimal Variance Sampling with Provable Guarantees for Fast Training of Graph Neural Networkscited by 54
  12. 2021Scalable and Adaptive Graph Neural Networks with Self-Label-Enhanced trainingcited by 54
  13. 2020Reducing Communication in Graph Neural Network Trainingcited by 50
  14. 2020Deep Graph Generators: A Surveycited by 36
  15. 2021A Biased Graph Neural Network Sampler with Near-Optimal Regretcited by 17
  16. 2021RA-GCN: Graph Convolutional Network for Disease Prediction Problems with Imbalanced Datacited by 14
  17. 2020Distributed Training of Graph Convolutional Networks using Subgraph Approximationcited by 7
  18. 2021Communication-Efficient Sampling for Distributed Training of Graph Convolutional Networkscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 10; 2024: 3; 2025: 3; 2026: 2.

24 papers.

Sort:

By area:

  1. 2022Distributed Graph Neural Network Training: A Surveycited by 32
  2. 2022FedGCN: Convergence-Communication Tradeoffs in Federated Training of Graph Convolutional Networkscited by 24
  3. 2023Efficiently Forgetting What You Have Learned in Graph Representation Learning via Projectioncited by 20
  4. 2023A Survey on Graph Neural Network Acceleration: Algorithms, Systems, and Customized Hardwarecited by 18
  5. 2022A Comprehensive Survey on Distributed Training of Graph Neural Networkscited by 15
  6. 2023Distill to Delete: Unlearning in Graph Networks with Knowledge Distillationcited by 13
  7. 2022RSC: Accelerating Graph Neural Networks Training via Randomized Sparse Computationscited by 12
  8. 2023Graph Coarsening via Convolution Matching for Scalable Graph Neural Network Trainingcited by 12
  9. 2023Simplifying Distributed Neural Network Training on Massive Graphs: Randomized Partitions Improve Model Aggregationcited by 7
  10. 2023FedGKD: Unleashing the Power of Collaboration in Federated Graph Neural Networkscited by 7
  11. 2023Staleness-Alleviated Distributed GNN Training via Online Dynamic-Embedding Predictioncited by 6
  12. 2024Multi-Modal Recommendation Unlearning for Legal, Licensing, and Modality Constraintscited by 5
12 more
  1. 2022Distributed Graph Neural Network Training with Periodic Stale Representation Synchronizationcited by 3
  2. 2023Boosting Distributed Full-graph GNN Training with Asynchronous One-bit Communicationcited by 3
  3. 2024MassiveGNN: Efficient Training via Prefetching for Massively Connected Distributed Graphscited by 3
  4. 2023Graph Ladling: Shockingly Simple Parallel GNN Training without Intermediate Communicationcited by 2
  5. 2023Privacy-preserving design of graph neural networks with applications to vertical federated learningcited by 2
  6. 2022DistGNN-MB: Distributed Large-Scale Graph Neural Network Training on x86 via Minibatch Samplingcited by 0
  7. 2024SDT-GNN: Streaming-based Distributed Training Framework for Graph Neural Networkscited by 0
  8. 2025MixGCN: Scalable GCN Training by Mixture of Parallelism and Mixture of Acceleratorscited by 0
  9. 2025Enhanced Soups for Graph Neural Networkscited by 0
  10. 2025Demystifying Distributed Training of Graph Neural Networks for Link Predictioncited by 0
  11. 2026DG-CoLearn: An Efficient Collaborative Learning Framework for Dynamic Graphscited by 0
  12. 2026Two-level domain-decomposition AdaGrad method for scalable training of graph neural networkscited by 0