# DistDGL: Distributed Graph Neural Network Training for Billion-Scale Graphs

arXiv: https://arxiv.org/abs/2010.05337 · HTML: https://mlsanity.com/paper/2010.05337/

## Key facts

- Cited by 139 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Oct 2020.

## Built on by (most cited first, 30 of 139)

- [Combinatorial Optimization with Physics-Inspired Graph Neural Networks](https://mlsanity.com/paper/2107.01188/) (2021)
- [GNNAutoScale: Scalable and Expressive Graph Neural Networks via Historical Embeddings](https://mlsanity.com/paper/2106.05609/) (2021)
- [Graph Attention Multi-Layer Perceptron](https://mlsanity.com/paper/2206.04355/) (2022)
- [TGL: A General Framework for Temporal GNN Training on Billion-Scale Graphs](https://mlsanity.com/paper/2203.14883/) (2022)
- [DistGNN: Scalable Distributed Training for Large-Scale Graph Neural Networks](https://mlsanity.com/paper/2104.06700/) (2021)
- [BGL: GPU-Efficient GNN Training by Optimizing Graph Data I/O and Preprocessing](https://mlsanity.com/paper/2112.08541/) (2021)
- [BNS-GCN: Efficient Full-Graph Training of Graph Convolutional Networks with Partition-Parallelism and Random Boundary Node Sampling](https://mlsanity.com/paper/2203.10983/) (2022)
- [Accelerating Training and Inference of Graph Neural Networks with Fast Sampling and Pipelining](https://mlsanity.com/paper/2110.08450/) (2021)
- [PaSca: a Graph Neural Architecture Search System under the Scalable Paradigm](https://mlsanity.com/paper/2203.00638/) (2022)
- [Distributed Graph Neural Network Training: A Survey](https://mlsanity.com/paper/2211.00216/) (2022)
- [IGB: Addressing The Gaps In Labeling, Features, Heterogeneity, and Size of Public Graph Datasets for Deep Learning Research](https://mlsanity.com/paper/2302.13522/) (2023)
- [Graph Coloring with Physics-Inspired Graph Neural Networks](https://mlsanity.com/paper/2202.01606/) (2022)
- [Parallel and Distributed Graph Neural Networks: An In-Depth Concurrency Analysis](https://mlsanity.com/paper/2205.09702/) (2022)
- [Train Your Own GNN Teacher: Graph-Aware Distillation on Textual Graphs](https://mlsanity.com/paper/2304.10668/) (2023)
- [Survey on Graph Neural Network Acceleration: An Algorithmic Perspective](https://mlsanity.com/paper/2202.04822/) (2022)
- [Large Graph Convolutional Network Training with GPU-Oriented Data Communication Architecture](https://mlsanity.com/paper/2103.03330/) (2021)
- [Towards Training Billion Parameter Graph Neural Networks for Atomic Simulations](https://mlsanity.com/paper/2203.09697/) (2022)
- [Learn Locally, Correct Globally: A Distributed Algorithm for Training Graph Neural Networks](https://mlsanity.com/paper/2111.08202/) (2021)
- [Global Neighbor Sampling for Mixed CPU-GPU Training on Giant Graphs](https://mlsanity.com/paper/2106.06150/) (2021)
- [TF-GNN: Graph Neural Networks in TensorFlow](https://mlsanity.com/paper/2207.03522/) (2022)
- [Ginex: SSD-enabled Billion-scale Graph Neural Network Training on a Single Machine via Provably Optimal In-memory Caching](https://mlsanity.com/paper/2208.09151/) (2022)
- [Evaluating Deep Graph Neural Networks](https://mlsanity.com/paper/2108.00955/) (2021)
- [MariusGNN: Resource-Efficient Out-of-Core Training of Graph Neural Networks](https://mlsanity.com/paper/2202.02365/) (2022)
- [Sequential Aggregation and Rematerialization: Distributed Full-batch Training of Graph Neural Networks on Large Graphs](https://mlsanity.com/paper/2111.06483/) (2021)
- [A Survey on Graph Neural Network Acceleration: Algorithms, Systems, and Customized Hardware](https://mlsanity.com/paper/2306.14052/) (2023)
- [Communication-Efficient Graph Neural Networks with Probabilistic Neighborhood Expansion Analysis and Caching](https://mlsanity.com/paper/2305.03152/) (2023)
- [A Comprehensive Survey on Distributed Training of Graph Neural Networks](https://mlsanity.com/paper/2211.05368/) (2022)
- [Mixture-of-Partitions: Infusing Large Biomedical Knowledge Graphs into BERT](https://mlsanity.com/paper/2109.04810/) (2021)
- [The Evolution of Distributed Systems for Graph Neural Networks and their Origin in Graph Processing and Deep Learning: A Survey](https://mlsanity.com/paper/2305.13854/) (2023)
- [SmartSAGE: Training Large-scale Graph Neural Networks using In-Storage Processing Architectures](https://mlsanity.com/paper/2205.04711/) (2022)

## Abstract (arXiv)

Graph neural networks (GNN) have shown great success in learning from graph-structured data. They are widely used in various applications, such as recommendation, fraud detection, and search. In these domains, the graphs are typically large, containing hundreds of millions of nodes and several billions of edges. To tackle this challenge, we develop DistDGL, a system for training GNNs in a mini-batch fashion on a cluster of machines. DistDGL is based on the Deep Graph Library (DGL), a popular GNN development framework. DistDGL distributes the graph and its associated data (initial features and embeddings) across the machines and uses this distribution to derive a computational decomposition by following an owner-compute rule. DistDGL follows a synchronous training approach and allows ego-networks forming the mini-batches to include non-local nodes. To minimize the overheads associated with distributed computations, DistDGL uses a high-quality and light-weight min-cut graph partitioning algorithm along with multiple balancing constraints. This allows it to reduce communication overheads and statically balance the computations. It further reduces the communication by replicating halo nodes and by using sparse embedding updates. The combination of these design choices allows DistDGL to train high-quality models while achieving high parallel efficiency and memory scalability. We demonstrate our optimizations on both inductive and transductive GNN models. Our results show that DistDGL achieves linear speedup without compromising model accuracy and requires only 13 seconds to complete a training epoch for a graph with 100 million nodes and 3 billion edges on a cluster with 16 machines. DistDGL is now publicly available as part of DGL:https://github.com/dmlc/dgl/tree/master/python/dgl/distributed.