# GraphTheta: A Distributed Graph Neural Network Learning System With Flexible Training Strategy arXiv: https://arxiv.org/abs/2104.10569 · HTML: https://mlsanity.com/paper/2104.10569/ ## Key facts - Cited by 8 papers in the map (data through 14 Oct 2026). - Builds on 48 of its 114 references that are in the map. - First posted on arXiv 21 Apr 2021. ## Builds on (most cited first, 30 of 48) - [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014) - [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019) - [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016) - [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017) - [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017) - [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018) - [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017) - [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019) - [Open Graph Benchmark: Datasets for Machine Learning on Graphs](https://mlsanity.com/paper/2005.00687/) (2020) - [Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering](https://mlsanity.com/paper/1606.09375/) (2016) - [Graph Convolutional Neural Networks for Web-Scale Recommender Systems](https://mlsanity.com/paper/1806.01973/) (2018) - [Representation Learning on Graphs with Jumping Knowledge Networks](https://mlsanity.com/paper/1806.03536/) (2018) - [Simple and Deep Graph Convolutional Networks](https://mlsanity.com/paper/2007.02133/) (2020) - [Link Prediction Based on Graph Neural Networks](https://mlsanity.com/paper/1802.09691/) (2018) - [GraphSAINT: Graph Sampling Based Inductive Learning Method](https://mlsanity.com/paper/1907.04931/) (2019) - [FastGCN: Fast Learning with Graph Convolutional Networks via Importance Sampling](https://mlsanity.com/paper/1801.10247/) (2018) - [Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks](https://mlsanity.com/paper/1905.07953/) (2019) - [DeepGCNs: Can GCNs Go as Deep as CNNs?](https://mlsanity.com/paper/1904.03751/) (2019) - [Representation Learning on Graphs: Methods and Applications](https://mlsanity.com/paper/1709.05584/) (2017) - [Deep Learning on Graphs: A Survey](https://mlsanity.com/paper/1812.04202/) (2018) - [OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphs](https://mlsanity.com/paper/2103.09430/) (2021) - [Hyperbolic Graph Neural Networks](https://mlsanity.com/paper/1910.12892/) (2019) - [Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networks](https://mlsanity.com/paper/1909.01315/) (2019) - [Learning to Represent Programs with Graphs](https://mlsanity.com/paper/1711.00740/) (2017) - [MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems](https://mlsanity.com/paper/1512.01274/) (2015) - [Adaptive Sampling Towards Fast Graph Representation Learning](https://mlsanity.com/paper/1809.05343/) (2018) - [Scaling Graph Neural Networks with Approximate PageRank](https://mlsanity.com/paper/2007.01570/) (2020) - [AliGraph: A Comprehensive Graph Neural Network Platform](https://mlsanity.com/paper/1902.08730/) (2019) - [DeepCoder: Learning to Write Programs](https://mlsanity.com/paper/1611.01989/) (2016) - [DistDGL: Distributed Graph Neural Network Training for Billion-Scale Graphs](https://mlsanity.com/paper/2010.05337/) (2020) ## Built on by (most cited first, 8 of 8) - [Distributed Graph Neural Network Training: A Survey](https://mlsanity.com/paper/2211.00216/) (2022) - [A Comprehensive Survey on Distributed Training of Graph Neural Networks](https://mlsanity.com/paper/2211.05368/) (2022) - [The Evolution of Distributed Systems for Graph Neural Networks and their Origin in Graph Processing and Deep Learning: A Survey](https://mlsanity.com/paper/2305.13854/) (2023) - [Towards high-accuracy deep learning inference of compressible turbulent flows over aerofoils](https://mlsanity.com/paper/2109.02183/) (2021) - [GraphRPM: Risk Pattern Mining on Industrial Large Attributed Graphs](https://mlsanity.com/paper/2411.06878/) (2024) - [Distributed Graph Neural Network Inference With Just-In-Time Compilation For Industry-Scale Graphs](https://mlsanity.com/paper/2503.06208/) (2025) - [GraphGen+: Advancing Distributed Subgraph Generation and Graph Learning On Industrial Graphs](https://mlsanity.com/paper/2503.06212/) (2025) - [GIPA: A General Information Propagation Algorithm for Graph Learning](https://mlsanity.com/paper/2301.08209/) (2023) ## Abstract (arXiv) Graph neural networks (GNNs) have been demonstrated as a powerful tool for analyzing non-Euclidean graph data. However, the lack of efficient distributed graph learning systems severely hinders applications of GNNs, especially when graphs are big and GNNs are relatively deep. Herein, we present GraphTheta, the first distributed and scalable graph learning system built upon vertex-centric distributed graph processing with neural network operators implemented as user-defined functions. This system supports multiple training strategies and enables efficient and scalable big-graph learning on distributed (virtual) machines with low memory. To facilitate graph convolutions, GraphTheta puts forward a new graph learning abstraction named NN-TGAR to bridge the gap between graph processing and graph deep learning. A distributed graph engine is proposed to conduct the stochastic gradient descent optimization with a hybrid-parallel execution, and a new cluster-batched training strategy is supported. We evaluate GraphTheta using several datasets with network sizes ranging from small-, modest- to large-scale. Experimental results show that GraphTheta can scale well to 1,024 workers for training an in-house developed GNN on an industry-scale Alipay dataset of 1.4 billion nodes and 4.1 billion attributed edges, with a cluster of CPU virtual machines (dockers) of small memory each (5~12GB). Moreover, GraphTheta can outperform DistDGL by up to 2.02×, with better scalability, and GraphLearn by up to 30.56×. As for model accuracy, GraphTheta is capable of learning as good GNNs as existing frameworks. To the best of our knowledge, this work presents the largest edge-attributed GNN learning task in the literature.