mlsanityData to 14 Oct 2026

GraNNDis: Efficient Unified Distributed Training Framework for Deep GNNs on Large Clusters

arXiv 2311.06837 ↗First posted 12 Nov 2023cs.LG cs.DCcited by 2

Not on the map. Why?

Abstract (arXiv)

Graph neural networks (GNNs) are one of the rapidly growing fields within deep learning. While many distributed GNN training frameworks have been proposed to increase the training throughput, they face three limitations when applied to multi-server clusters. 1) They suffer from an inter-server communication bottleneck because they do not consider the inter-/intra-server bandwidth gap, a representative characteristic of multi-server clusters. 2) Redundant memory usage and computation hinder the scalability of the distributed frameworks. 3) Sampling methods, de facto standard in mini-batch training, incur unnecessary errors in multi-server clusters. We found that these limitations can be addressed by exploiting the characteristics of multi-server clusters. Here, we propose GraNNDis, a fast distributed GNN training framework for multi-server clusters. Firstly, we present Flexible Preloading, which preloads the essential vertex dependencies server-wise to reduce the low-bandwidth inter-server communications. Secondly, we introduce Cooperative Batching, which enables memory-efficient, less redundant mini-batch training by utilizing high-bandwidth intra-server communications. Thirdly, we propose Expansion-aware Sampling, a cluster-aware sampling method, which samples the edges that affect the system speedup. As sampling the intra-server dependencies does not contribute much to the speedup as they are communicated through fast intra-server links, it only targets a server boundary to be sampled. Lastly, we introduce One-Hop Graph Masking, a computation and communication structure to realize the above methods in multi-server environments. We evaluated GraNNDis on multi-server clusters, and it provided significant speedup over the state-of-the-art distributed GNN training frameworks. GraNNDis is open-sourced at https://github.com/AIS-SNU/GraNNDis_Artifact to facilitate its use.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 45 of its 96 references that are in the map. First posted on arXiv 12 Nov 2023.

Builds on

45 of 96 references.

References · 45

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2017Graph Attention Networkscited by 8,923
  4. 2017Inductive Representation Learning on Large Graphscited by 6,320
  5. 2018How Powerful are Graph Neural Networks?cited by 4,393
  6. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  7. 2019Fast Graph Representation Learning with PyTorch Geometriccited by 2,434
  8. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,163
  9. 2016Convolutional Neural Networks on Graphs with Fast Localized Spectral Filteringcited by 2,089
  10. 2013Spectral Networks and Locally Connected Networks on Graphscited by 1,102
  11. 2019Graph Neural Networks for Social Recommendationcited by 714
  12. 2021E(3)-Equivariant Graph Neural Networks for Data-Efficient and Accurate Interatomic Potentialscited by 708
33 more
  1. 2019DropEdge: Towards Deep Graph Convolutional Networks on Node Classificationcited by 680
  2. 2019GraphSAINT: Graph Sampling Based Inductive Learning Methodcited by 618
  3. 2018FastGCN: Fast Learning with Graph Convolutional Networks via Importance Samplingcited by 584
  4. 2019Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networkscited by 541
  5. 2021GemNet: Universal Directional Graph Neural Networks for Moleculescited by 343
  6. 2019Multi-Label Image Recognition with Graph Convolutional Networkscited by 322
  7. 2021Graph Neural Network-Based Anomaly Detection in Multivariate Time Seriescited by 301
  8. 2022Equiformer: Equivariant Graph Attention Transformer for 3D Atomistic Graphscited by 264
  9. 2020DeeperGCN: All You Need to Train Deeper GCNscited by 250
  10. 2022Representing Long-Range Context for Graph Neural Networks with Global Attentioncited by 195
  11. 2021Training Graph Neural Networks with 1000 Layerscited by 169
  12. 2019AliGraph: A Comprehensive Graph Neural Network Platformcited by 153
  13. 2020DistDGL: Distributed Graph Neural Network Training for Billion-Scale Graphscited by 139
  14. 2021GNNAutoScale: Scalable and Expressive Graph Neural Networks via Historical Embeddingscited by 90
  15. 2020HyGCN: A GCN Accelerator with Hybrid Architecturecited by 74
  16. 2020Reducing Communication in Graph Neural Network Trainingcited by 50
  17. 2021BGL: GPU-Efficient GNN Training by Optimizing Graph Data I/O and Preprocessingcited by 50
  18. 2022PipeGCN: Efficient Full-Graph Training of Graph Convolutional Networks with Pipelined Feature Communicationcited by 45
  19. 2022BNS-GCN: Efficient Full-Graph Training of Graph Convolutional Networks with Partition-Parallelism and Random Boundary Node Samplingcited by 43
  20. 2019Evaluating Modern GPU Interconnect: PCIe, NVLink, NV-SLI, NVSwitch and GPUDirectcited by 41
  21. 2019Syntax-Aware Aspect Level Sentiment Classification with Graph Attention Networkscited by 40
  22. 2021Accelerating Training and Inference of Graph Neural Networks with Fast Sampling and Pipeliningcited by 39
  23. 2021Distributed Hybrid CPU and GPU training for Graph Neural Networks on Billion-Scale Graphscited by 33
  24. 2023IGB: Addressing The Gaps In Labeling, Features, Heterogeneity, and Size of Public Graph Datasets for Deep Learning Researchcited by 30
  25. 2023LMC: Fast Training of GNNs via Subgraph Sampling with Provable Convergencecited by 26
  26. 2021Large Graph Convolutional Network Training with GPU-Oriented Data Communication Architecturecited by 25
  27. 2021Global Neighbor Sampling for Mixed CPU-GPU Training on Giant Graphscited by 22
  28. 2022Ginex: SSD-enabled Billion-scale Graph Neural Network Training on a Single Machine via Provably Optimal In-memory Cachingcited by 21
  29. 2023Communication-Efficient Graph Neural Networks with Probabilistic Neighborhood Expansion Analysis and Cachingcited by 16
  30. 2021Scalable Graph Neural Network Training: The Case for Samplingcited by 11
  31. 2023Quiver: Supporting GPUs for Low-Latency, High-Throughput GNN Serving with Workload Awarenesscited by 11
  32. 2021GNNear: Accelerating Full-Batch Training of Graph Neural Networks with Near-Memory Processingcited by 6
  33. 2023Helios: An Efficient Out-of-core GNN Training System on Terabyte-scale Graphs with In-memory Performancecited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026FeLoG: Scalable and Efficient Distributed Graph Embedding with Feedback Loop Mechanismcited by 0
  2. 2026SNI-GNN: SmartNIC-Assisted Full-Graph GNN Training with In-Network Embedding Predictioncited by 0