mlsanityData to 14 Oct 2026

DistGNN: Scalable Distributed Training for Large-Scale Graph Neural Networks

arXiv 2104.06700 ↗First posted 14 Apr 2021cs.LG cs.DCcited by 59

Not on the map. Why?

Abstract (arXiv)

Full-batch training on Graph Neural Networks (GNN) to learn the structure of large graphs is a critical problem that needs to scale to hundreds of compute nodes to be feasible. It is challenging due to large memory capacity and bandwidth requirements on a single compute node and high communication volumes across multiple nodes. In this paper, we present DistGNN that optimizes the well-known Deep Graph Library (DGL) for full-batch training on CPU clusters via an efficient shared memory implementation, communication reduction using a minimum vertex-cut graph partitioning algorithm and communication avoidance using a family of delayed-update algorithms. Our results on four common GNN benchmark datasets: Reddit, OGB-Products, OGB-Papers and Proteins, show up to 3.7x speed-up using a single CPU socket and up to 97x speed-up using 128 CPU sockets, respectively, over baseline DGL implementations running on a single CPU socket

Key facts. Cited by 59 papers in the map (data through 14 Oct 2026). Builds on 19 of its 36 references that are in the map. First posted on arXiv 14 Apr 2021.

Builds on

19 of 36 references.

References · 19

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2017Inductive Representation Learning on Large Graphscited by 6,320
  3. 2018How Powerful are Graph Neural Networks?cited by 4,393
  4. 2019Fast Graph Representation Learning with PyTorch Geometriccited by 2,434
  5. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,163
  6. 2018Graph Convolutional Neural Networks for Web-Scale Recommender Systemscited by 1,138
  7. 2016Geometric deep learning: going beyond Euclidean datacited by 1,048
  8. 2018FastGCN: Fast Learning with Graph Convolutional Networks via Importance Samplingcited by 584
  9. 2019Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networkscited by 541
  10. 2020Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classificationcited by 506
  11. 2018Modeling polypharmacy side effects with graph convolutional networkscited by 237
  12. 2015MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systemscited by 195
7 more
  1. 2019AliGraph: A Comprehensive Graph Neural Network Platformcited by 153
  2. 2020DistDGL: Distributed Graph Neural Network Training for Billion-Scale Graphscited by 139
  3. 2018Embedding Logical Queries on Knowledge Graphscited by 120
  4. 2020Reducing Communication in Graph Neural Network Trainingcited by 50
  5. 2020FeatGraph: A Flexible and Efficient Backend for Graph Neural Network Systemscited by 36
  6. 2020Deep Graph Library Optimizations for Intel(R) x86 Architecturecited by 3
  7. 2020A Vertex Cut based Framework for Load Balancing and Parallelism Optimization in Multi-core Systemscited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 5; 2022: 15; 2023: 14; 2024: 15; 2025: 9; 2026: 1.

59 papers.

Sort:

  1. 2022A Comprehensive Study on Large-Scale Graph Training: Benchmarking and Rethinkingcited by 56
  2. 2022Distributed Graph Neural Network Training: A Surveycited by 32
  3. 2022Parallel and Distributed Graph Neural Networks: An In-Depth Concurrency Analysiscited by 29
  4. 2022Survey on Graph Neural Network Acceleration: An Algorithmic Perspectivecited by 26
  5. 2021Learn Locally, Correct Globally: A Distributed Algorithm for Training Graph Neural Networkscited by 24
  6. 2022Scalable algorithms for physics-informed neural and graph networkscited by 21
  7. 2024Anomaly Detection in Dynamic Graphs: A Comprehensive Surveycited by 19
  8. 2021Sequential Aggregation and Rematerialization: Distributed Full-batch Training of Graph Neural Networks on Large Graphscited by 18
  9. 2023Communication-Efficient Graph Neural Networks with Probabilistic Neighborhood Expansion Analysis and Cachingcited by 16
  10. 2022A Comprehensive Survey on Distributed Training of Graph Neural Networkscited by 15
  11. 2022The Open MatSci ML Toolkit: A Flexible Framework for Machine Learning in Materials Sciencecited by 14
  12. 2023The Evolution of Distributed Systems for Graph Neural Networks and their Origin in Graph Processing and Deep Learning: A Surveycited by 14
38 more
  1. 2022RSC: Accelerating Graph Neural Networks Training via Randomized Sparse Computationscited by 12
  2. 2024Graph Neural Network Training Systems: A Performance Comparison of Full-Graph and Mini-Batchcited by 10
  3. 2021GraphTheta: A Distributed Graph Neural Network Learning System With Flexible Training Strategycited by 8
  4. 2023Enabling Resource-efficient AIoT System with Cross-level Optimization: A surveycited by 8
  5. 2024Scalable Training of Trustworthy and Energy-Efficient Predictive Graph Foundation Models for Atomistic Materials Modeling: A Case Study with HydraGNNcited by 8
  6. 2025GiGL: Large-Scale Graph Neural Networks at Snapchatcited by 8
  7. 2022Characterizing and Understanding Distributed GNN Training on GPUscited by 7
  8. 2023Simplifying Distributed Neural Network Training on Massive Graphs: Randomized Partitions Improve Model Aggregationcited by 7
  9. 2023SPEED: Streaming Partition and Parallel Acceleration for Temporal Interaction Graph Embeddingcited by 7
  10. 2021GNNear: Accelerating Full-Batch Training of Graph Neural Networks with Near-Memory Processingcited by 6
  11. 2023GSplit: Scaling Graph Neural Network Training on Large Graphs via Split-Parallelismcited by 6
  12. 2023Staleness-Alleviated Distributed GNN Training via Online Dynamic-Embedding Predictioncited by 6
  13. 2024Generative Active Learning for the Search of Small-molecule Protein Binderscited by 6
  14. 2025Fast and Distributed Equivariant Graph Neural Networks by Virtual Node Learningcited by 6
  15. 2022Scaling R-GCN Training with Graph Summarizationcited by 5
  16. 2021MG-GCN: Scalable Multi-GPU GCN Training Frameworkcited by 4
  17. 2023FreshGNN: Reducing Memory Access via Stable Historical Embeddings for Graph Neural Network Trainingcited by 4
  18. 2023Tango: rethinking quantization for graph neural network training on GPUscited by 4
  19. 2023An Experimental Comparison of Partitioning Strategies for Distributed Graph Neural Network Trainingcited by 4
  20. 2024Survey of Graph Neural Network for Internet of Things and NextG Networkscited by 4
  21. 2024Demystifying Higher-Order Graph Neural Networkscited by 4
  22. 2024Can Graph Reordering Speed Up Graph Neural Network Training? An Experimental Studycited by 4
  23. 2022Characterizing the Efficiency of Graph Neural Network Frameworks with a Magnifying Glasscited by 3
  24. 2024PyGim: An Efficient Graph Neural Network Library for Real Processing-In-Memory Architecturescited by 3
  25. 2024Input Snapshots Fusion for Scalable Discrete-Time Dynamic Graph Neural Networkscited by 3
  26. 2024Edge Graph Intelligence: Reciprocally Empowering Edge Networks with Graph Intelligencecited by 3
  27. 2025Sparsity-Aware Communication for Distributed Graph Neural Network Trainingcited by 3
  28. 2023BatchGNN: Efficient CPU-Based Distributed GNN Training on Very Large Graphscited by 2
  29. 2024Graph Neural Networks Gone Hogwildcited by 2
  30. 2025Reforge: Low-Latency Distributed GNN Serving with Selective Embedding Recomputationcited by 2
  31. 2022FPGA-based AI Smart NICs for Scalable Distributed AI Training Systemscited by 1
  32. 2023DGC: Training Dynamic Graphs with Spatio-Temporal Non-Uniformity using Graph Partitioning by Chunkscited by 1
  33. 2024Single-GPU GNN Systems: Traps and Pitfallscited by 1
  34. 2025Deal: Distributed End-to-End GNN Inference for All Nodescited by 1
  35. 2025Multi-task parallelism for robust pre-training of graph foundation models on multi-source, multi-fidelity atomistic modeling datacited by 1
  36. 2022Scaling Knowledge Graph Embedding Modelscited by 0
  37. 2022Analysis and Optimization of GNN-Based Recommender Systems on Persistent Memorycited by 0
  38. 2022Architectural Implications of Embedding Dimension during GCN on CPU and GPUcited by 0

50 most cited of 59 All 59 (JSON)