# Learn Locally, Correct Globally: A Distributed Algorithm for Training Graph Neural Networks

arXiv: https://arxiv.org/abs/2111.08202 · HTML: https://mlsanity.com/paper/2111.08202/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 43 references that are in the map.
- First posted on arXiv 16 Nov 2021.

## Builds on (most cited first, 30 of 30)

- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [Open Graph Benchmark: Datasets for Machine Learning on Graphs](https://mlsanity.com/paper/2005.00687/) (2020)
- [Federated Learning: Strategies for Improving Communication Efficiency](https://mlsanity.com/paper/1610.05492/) (2016)
- [On the Convergence of FedAvg on Non-IID Data](https://mlsanity.com/paper/1907.02189/) (2019)
- [Graph Convolutional Neural Networks for Web-Scale Recommender Systems](https://mlsanity.com/paper/1806.01973/) (2018)
- [Predict then Propagate: Graph Neural Networks meet Personalized PageRank](https://mlsanity.com/paper/1810.05997/) (2018)
- [Towards Federated Learning at Scale: System Design](https://mlsanity.com/paper/1902.01046/) (2019)
- [GraphSAINT: Graph Sampling Based Inductive Learning Method](https://mlsanity.com/paper/1907.04931/) (2019)
- [Local SGD Converges Fast and Communicates Little](https://mlsanity.com/paper/1805.09767/) (2018)
- [Federated Learning for Mobile Keyboard Prediction](https://mlsanity.com/paper/1811.03604/) (2018)
- [Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks](https://mlsanity.com/paper/1905.07953/) (2019)
- [DeepGCNs: Can GCNs Go as Deep as CNNs?](https://mlsanity.com/paper/1904.03751/) (2019)
- [OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphs](https://mlsanity.com/paper/2103.09430/) (2021)
- [Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learning](https://mlsanity.com/paper/1807.06629/) (2018)
- [Layer-Dependent Importance Sampling for Training Deep and Large Graph Convolutional Networks](https://mlsanity.com/paper/1911.07323/) (2019)
- [DistDGL: Distributed Graph Neural Network Training for Billion-Scale Graphs](https://mlsanity.com/paper/2010.05337/) (2020)
- [Billion-scale Commodity Embedding for E-commerce Recommendation in Alibaba](https://mlsanity.com/paper/1803.02349/) (2018)
- [DistGNN: Scalable Distributed Training for Large-Scale Graph Neural Networks](https://mlsanity.com/paper/2104.06700/) (2021)
- [Graph Transformation Policy Network for Chemical Reaction Prediction](https://mlsanity.com/paper/1812.09441/) (2018)
- [Minimal Variance Sampling with Provable Guarantees for Fast Training of Graph Neural Networks](https://mlsanity.com/paper/2006.13866/) (2020)
- [Scalable and Adaptive Graph Neural Networks with Self-Label-Enhanced training](https://mlsanity.com/paper/2104.09376/) (2021)
- [Reducing Communication in Graph Neural Network Training](https://mlsanity.com/paper/2005.03300/) (2020)
- [Deep Graph Generators: A Survey](https://mlsanity.com/paper/2012.15544/) (2020)
- [A Biased Graph Neural Network Sampler with Near-Optimal Regret](https://mlsanity.com/paper/2103.01089/) (2021)
- [RA-GCN: Graph Convolutional Network for Disease Prediction Problems with Imbalanced Data](https://mlsanity.com/paper/2103.00221/) (2021)
- [Distributed Training of Graph Convolutional Networks using Subgraph Approximation](https://mlsanity.com/paper/2012.04930/) (2020)
- [Communication-Efficient Sampling for Distributed Training of Graph Convolutional Networks](https://mlsanity.com/paper/2101.07706/) (2021)

## Built on by (most cited first, 24 of 24)

- [Distributed Graph Neural Network Training: A Survey](https://mlsanity.com/paper/2211.00216/) (2022)
- [FedGCN: Convergence-Communication Tradeoffs in Federated Training of Graph Convolutional Networks](https://mlsanity.com/paper/2201.12433/) (2022)
- [Efficiently Forgetting What You Have Learned in Graph Representation Learning via Projection](https://mlsanity.com/paper/2302.08990/) (2023)
- [A Survey on Graph Neural Network Acceleration: Algorithms, Systems, and Customized Hardware](https://mlsanity.com/paper/2306.14052/) (2023)
- [A Comprehensive Survey on Distributed Training of Graph Neural Networks](https://mlsanity.com/paper/2211.05368/) (2022)
- [Distill to Delete: Unlearning in Graph Networks with Knowledge Distillation](https://mlsanity.com/paper/2309.16173/) (2023)
- [RSC: Accelerating Graph Neural Networks Training via Randomized Sparse Computations](https://mlsanity.com/paper/2210.10737/) (2022)
- [Graph Coarsening via Convolution Matching for Scalable Graph Neural Network Training](https://mlsanity.com/paper/2312.15520/) (2023)
- [Simplifying Distributed Neural Network Training on Massive Graphs: Randomized Partitions Improve Model Aggregation](https://mlsanity.com/paper/2305.09887/) (2023)
- [FedGKD: Unleashing the Power of Collaboration in Federated Graph Neural Networks](https://mlsanity.com/paper/2309.09517/) (2023)
- [Staleness-Alleviated Distributed GNN Training via Online Dynamic-Embedding Prediction](https://mlsanity.com/paper/2308.13466/) (2023)
- [Multi-Modal Recommendation Unlearning for Legal, Licensing, and Modality Constraints](https://mlsanity.com/paper/2405.15328/) (2024)
- [Distributed Graph Neural Network Training with Periodic Stale Representation Synchronization](https://mlsanity.com/paper/2206.00057/) (2022)
- [Boosting Distributed Full-graph GNN Training with Asynchronous One-bit Communication](https://mlsanity.com/paper/2303.01277/) (2023)
- [MassiveGNN: Efficient Training via Prefetching for Massively Connected Distributed Graphs](https://mlsanity.com/paper/2410.22697/) (2024)
- [Graph Ladling: Shockingly Simple Parallel GNN Training without Intermediate Communication](https://mlsanity.com/paper/2306.10466/) (2023)
- [Privacy-preserving design of graph neural networks with applications to vertical federated learning](https://mlsanity.com/paper/2310.20552/) (2023)
- [DistGNN-MB: Distributed Large-Scale Graph Neural Network Training on x86 via Minibatch Sampling](https://mlsanity.com/paper/2211.06385/) (2022)
- [SDT-GNN: Streaming-based Distributed Training Framework for Graph Neural Networks](https://mlsanity.com/paper/2404.02300/) (2024)
- [MixGCN: Scalable GCN Training by Mixture of Parallelism and Mixture of Accelerators](https://mlsanity.com/paper/2501.01951/) (2025)
- [Enhanced Soups for Graph Neural Networks](https://mlsanity.com/paper/2503.11612/) (2025)
- [Demystifying Distributed Training of Graph Neural Networks for Link Prediction](https://mlsanity.com/paper/2506.20818/) (2025)
- [DG-CoLearn: An Efficient Collaborative Learning Framework for Dynamic Graphs](https://mlsanity.com/paper/2605.31427/) (2026)
- [Two-level domain-decomposition AdaGrad method for scalable training of graph neural networks](https://mlsanity.com/paper/2608.22575/) (2026)

## Abstract (arXiv)

Despite the recent success of Graph Neural Networks (GNNs), training GNNs on large graphs remains challenging. The limited resource capacities of the existing servers, the dependency between nodes in a graph, and the privacy concern due to the centralized storage and model learning have spurred the need to design an effective distributed algorithm for GNN training. However, existing distributed GNN training methods impose either excessive communication costs or large memory overheads that hinders their scalability. To overcome these issues, we propose a communication-efficient distributed GNN training technique named textLearn Locally, Correct Globally (LLCG). To reduce the communication and memory overhead, each local machine in LLCG first trains a GNN on its local data by ignoring the dependency between nodes among different machines, then sends the locally trained model to the server for periodic model averaging. However, ignoring node dependency could result in significant performance degradation. To solve the performance degradation, we propose to apply textGlobal Server Corrections on the server to refine the locally learned models. We rigorously analyze the convergence of distributed methods with periodic model averaging for training GNNs and show that naively applying periodic model averaging but ignoring the dependency between nodes will suffer from an irreducible residual error. However, this residual error can be eliminated by utilizing the proposed global corrections to entail fast convergence rate. Extensive experiments on real-world datasets show that LLCG can significantly improve the efficiency without hurting the performance.