mlsanityData to 7 Oct 2026

Distributed Hybrid CPU and GPU training for Graph Neural Networks on Billion-Scale Graphs

arXiv 2112.15345 ↗First posted 31 Dec 2021cs.DCcited by 33

Not on the map. Why?

Abstract (arXiv)

Graph neural networks (GNN) have shown great success in learning from graph-structured data. They are widely used in various applications, such as recommendation, fraud detection, and search. In these domains, the graphs are typically large and heterogeneous, containing many millions or billions of vertices and edges of different types. To tackle this challenge, we develop DistDGLv2, a system that extends DistDGL for training GNNs on massive heterogeneous graphs in a mini-batch fashion, using distributed hybrid CPU/GPU training. DistDGLv2 places graph data in distributed CPU memory and performs mini-batch computation in GPUs. For ease of use, DistDGLv2 adopts API compatible with Deep Graph Library (DGL)'s mini-batch training and heterogeneous graph API, which enables distributed training with almost no code modification. To ensure model accuracy, DistDGLv2 follows a synchronous training approach and allows ego-networks forming mini-batches to include non-local vertices. To ensure data locality and load balancing, DistDGLv2 partitions heterogeneous graphs by using a multi-level partitioning algorithm with min-edge cut and multiple balancing constraints. DistDGLv2 deploys an asynchronous mini-batch generation pipeline that makes computation and data access asynchronous to fully utilize all hardware (CPU, GPU, network, PCIe). The combination allows DistDGLv2 to train high-quality models while achieving high parallel efficiency and memory scalability. We demonstrate DistDGLv2 on various GNN workloads. Our results show that DistDGLv2 achieves 2-3X speedup over DistDGL and 18X speedup over Euler. It takes only 5-10 seconds to complete an epoch on graphs with hundreds of millions of vertices on a cluster with 64 GPUs.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). First posted on arXiv 31 Dec 2021.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 8; 2023: 6; 2024: 5; 2025: 9; 2026: 5.

33 papers.

Sort:

  1. 2022Distributed Graph Neural Network Training: A Surveycited by 32
  2. 2022Ginex: SSD-enabled Billion-scale Graph Neural Network Training on a Single Machine via Provably Optimal In-memory Cachingcited by 21
  3. 2022MariusGNN: Resource-Efficient Out-of-Core Training of Graph Neural Networkscited by 19
  4. 2022Layer-Neighbor Sampling -- Defusing Neighborhood Explosion in GNNscited by 19
  5. 2023A Survey on Graph Neural Network Acceleration: Algorithms, Systems, and Customized Hardwarecited by 18
  6. 2023Communication-Efficient Graph Neural Networks with Probabilistic Neighborhood Expansion Analysis and Cachingcited by 16
  7. 2022A Comprehensive Survey on Distributed Training of Graph Neural Networkscited by 15
  8. 2023DistTGL: Distributed Memory-Based Temporal Graph Neural Network Trainingcited by 12
  9. 2022Neural Graph Databasescited by 11
  10. 2024Graph Neural Network Training Systems: A Performance Comparison of Full-Graph and Mini-Batchcited by 10
  11. 2022Extreme Acceleration of Graph Neural Network-based Prediction Models for Quantum Chemistrycited by 3
  12. 2022Scalable Graph Convolutional Network Training on Distributed-Memory Systemscited by 2
21 more
  1. 2023BatchGNN: Efficient CPU-Based Distributed GNN Training on Very Large Graphscited by 2
  2. 2023GraNNDis: Efficient Unified Distributed Training Framework for Deep GNNs on Large Clusterscited by 2
  3. 2024D3-GNN: Dynamic Distributed Dataflow for Streaming Graph Neural Networkscited by 2
  4. 2025Reforge: Low-Latency Distributed GNN Serving with Selective Embedding Recomputationcited by 2
  5. 2025Partition-wise Graph Filtering: A Unified Perspective Through the Lens of Graph Coarseningcited by 2
  6. 2024GLISP: A Scalable GNN Learning System by Exploiting Inherent Structural Properties of Graphscited by 1
  7. 2024Characterizing and Understanding HGNN Training on GPUscited by 1
  8. 2025Deal: Distributed End-to-End GNN Inference for All Nodescited by 1
  9. 2025Asynchronous Sharpness-Aware Minimization For Fast and Accurate Deep Learningcited by 1
  10. 2025Multi-task parallelism for robust pre-training of graph foundation models on multi-source, multi-fidelity atomistic modeling datacited by 1
  11. 2023Synergistic Signals: Exploiting Co-Engagement and Semantic Links via Graph Neural Networkscited by 0
  12. 2024SDT-GNN: Streaming-based Distributed Training Framework for Graph Neural Networkscited by 0
  13. 2025Armada: Memory-Efficient Distributed Training of Large-Scale Graph Neural Networkscited by 0
  14. 2025WaveGAS: Waveform Relaxation for Scaling Graph Neural Networkscited by 0
  15. 2025MLKV: Efficiently Scaling up Large Embedding Model Training with Disk-based Key-Value Storagecited by 0
  16. 2025Demystifying Distributed Training of Graph Neural Networks for Link Predictioncited by 0
  17. 2026Accelerating Storage-Based Training for Graph Neural Networkscited by 0
  18. 2026MQ-GNN: A Multi-Queue Pipelined Architecture for Scalable and Efficient GNN Trainingcited by 0
  19. 2026Not All Neighbors Matter: Understanding the Impact of Graph Sparsification on GNN Pipelinescited by 0
  20. 2026Scalable and Adaptive Parallel Training of Graph Transformer on Large Graphscited by 0
  21. 2026SNI-GNN: SmartNIC-Assisted Full-Graph GNN Training with In-Network Embedding Predictioncited by 0