mlsanityData to 7 Oct 2026

Reducing Communication in Graph Neural Network Training

arXiv 2005.03300 ↗First posted 7 May 2020cs.LG cs.DC stat.MLcited by 50

Not on the map. Why?

Abstract (arXiv)

Graph Neural Networks (GNNs) are powerful and flexible neural networks that use the naturally sparse connectivity information of the data. GNNs represent this connectivity as sparse matrices, which have lower arithmetic intensity and thus higher communication costs compared to dense matrices, making GNNs harder to scale to high concurrencies than convolutional or fully-connected neural networks. We introduce a family of parallel algorithms for training GNNs and show that they can asymptotically reduce communication compared to previous parallel GNN training methods. We implement these algorithms, which are based on 1D, 1.5D, 2D, and 3D sparse-dense matrix multiplication, using torch.distributed on GPU-equipped clusters. Our algorithms optimize communication across the full GNN training pipeline. We train GNNs on over a hundred GPUs on multiple datasets, including a protein network with over a billion edges.

Key facts. Cited by 50 papers in the map (data through 7 Oct 2026). First posted on arXiv 7 May 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 14; 2023: 14; 2024: 5; 2025: 6; 2026: 2.

50 papers.

Sort:

By area:

  1. 2021DistGNN: Scalable Distributed Training for Large-Scale Graph Neural Networkscited by 59
  2. 2023A Hybrid Tensor-Expert-Data Parallelism Approach to Optimize Mixture-of-Experts Trainingcited by 47
  3. 2022PipeGCN: Efficient Full-Graph Training of Graph Convolutional Networks with Pipelined Feature Communicationcited by 45
  4. 2022BNS-GCN: Efficient Full-Graph Training of Graph Convolutional Networks with Partition-Parallelism and Random Boundary Node Samplingcited by 43
  5. 2022Distributed Graph Neural Network Training: A Surveycited by 32
  6. 2022Parallel and Distributed Graph Neural Networks: An In-Depth Concurrency Analysiscited by 29
  7. 2022Towards Training Billion Parameter Graph Neural Networks for Atomic Simulationscited by 25
  8. 2021Learn Locally, Correct Globally: A Distributed Algorithm for Training Graph Neural Networkscited by 24
  9. 2022Scalable algorithms for physics-informed neural and graph networkscited by 21
  10. 2021Sequential Aggregation and Rematerialization: Distributed Full-batch Training of Graph Neural Networks on Large Graphscited by 18
  11. 2023A Survey on Graph Neural Network Acceleration: Algorithms, Systems, and Customized Hardwarecited by 18
  12. 2023LazyGNN: Large-Scale Graph Neural Networks via Lazy Propagationcited by 16
38 more
  1. 2022A Comprehensive Survey on Distributed Training of Graph Neural Networkscited by 15
  2. 2022Graph Neural Networks in Particle Physics: Implementations, Innovations, and Challengescited by 12
  3. 2021Efficient Scaling of Dynamic Graph Neural Networkscited by 11
  4. 2022Neural Graph Databasescited by 11
  5. 2022GNN at the Edge: Cost-Efficient Graph Neural Network Processing over Distributed Edge Serverscited by 11
  6. 2023Adaptive Message Quantization and Parallelization for Distributed Full-graph GNN Trainingcited by 10
  7. 2023HOT: Higher-Order Dynamic Graph Representation Learning with Efficient Transformerscited by 10
  8. 2021GraphTheta: A Distributed Graph Neural Network Learning System With Flexible Training Strategycited by 8
  9. 2023Simplifying Distributed Neural Network Training on Massive Graphs: Randomized Partitions Improve Model Aggregationcited by 7
  10. 2021GNNear: Accelerating Full-Batch Training of Graph Neural Networks with Near-Memory Processingcited by 6
  11. 2023A 4D Hybrid Algorithm to Scale Parallel Training to Thousands of GPUscited by 6
  12. 2023Staleness-Alleviated Distributed GNN Training via Online Dynamic-Embedding Predictioncited by 6
  13. 2023Distributed Matrix-Based Sampling for Graph Neural Network Trainingcited by 6
  14. 2021GMLP: Building Scalable and Flexible Graph Neural Networks with Feature-Message Passingcited by 5
  15. 2021MG-GCN: Scalable Multi-GPU GCN Training Frameworkcited by 4
  16. 2024Demystifying Higher-Order Graph Neural Networkscited by 4
  17. 2025Subsampling Graphs with GNN Performance Guaranteescited by 4
  18. 2022Distributed-Memory Sparse Kernels for Machine Learningcited by 3
  19. 2022Distributed Graph Neural Network Training with Periodic Stale Representation Synchronizationcited by 3
  20. 2022Extreme Acceleration of Graph Neural Network-based Prediction Models for Quantum Chemistrycited by 3
  21. 2023Boosting Distributed Full-graph GNN Training with Asynchronous One-bit Communicationcited by 3
  22. 2025Sparsity-Aware Communication for Distributed Graph Neural Network Trainingcited by 3
  23. 2022Scalable Graph Convolutional Network Training on Distributed-Memory Systemscited by 2
  24. 2023BatchGNN: Efficient CPU-Based Distributed GNN Training on Very Large Graphscited by 2
  25. 2023GraNNDis: Efficient Unified Distributed Training Framework for Deep GNNs on Large Clusterscited by 2
  26. 2024PyGim: An Efficient Graph Neural Network Library for Real Processing-In-Memory Architecturescited by 2
  27. 2024CDFGNN: a Systematic Design of Cache-based Distributed Full-Batch Graph Neural Network Training with Communication Reductioncited by 2
  28. 2025GraphEdge: Dynamic Graph Partition and Task Scheduling for GNNs Computing in Edge Networkcited by 2
  29. 2021Tackling the Local Bias in Federated Graph Learningcited by 1
  30. 2023DGC: Training Dynamic Graphs with Spatio-Temporal Non-Uniformity using Graph Partitioning by Chunkscited by 1
  31. 2024GLISP: A Scalable GNN Learning System by Exploiting Inherent Structural Properties of Graphscited by 1
  32. 2024Single-GPU GNN Systems: Traps and Pitfallscited by 1
  33. 2025Deal: Distributed End-to-End GNN Inference for All Nodescited by 1
  34. 2023Cooperative Minibatching in Graph Neural Networkscited by 0
  35. 2025MixGCN: Scalable GCN Training by Mixture of Parallelism and Mixture of Acceleratorscited by 0
  36. 2025Adaptive Graph Pruning with Sudden-Events Evaluation for Traffic Prediction using Online Semi-Decentralized ST-GNNscited by 0
  37. 2026Communication-Avoiding Linear Algebraic Kernel K-Means on GPUscited by 0
  38. 2026Communication-free Sampling and 4D Hybrid Parallelism for Scalable Mini-batch GNN Trainingcited by 0