mlsanityData to 7 Oct 2026

GIST: Distributed Training for Large-Scale Graph Convolutional Networks

arXiv 2102.10424 ↗First posted 20 Feb 2021cs.LG cs.AI cs.DCcited by 10

Not on the map. Why?

Abstract (arXiv)

The graph convolutional network (GCN) is a go-to solution for machine learning on graphs, but its training is notoriously difficult to scale both in terms of graph size and the number of model parameters. Although some work has explored training on large-scale graphs (e.g., GraphSAGE, ClusterGCN, etc.), we pioneer efficient training of large-scale GCN models (i.e., ultra-wide, overparameterized models) with the proposal of a novel, distributed training framework. Our proposed training methodology, called GIST, disjointly partitions the parameters of a GCN model into several, smaller sub-GCNs that are trained independently and in parallel. In addition to being compatible with all GCN architectures and existing sampling techniques for efficient GCN training, GIST i) improves model performance, ii) scales to training on arbitrarily large graphs, iii) decreases wall-clock training time, and iv) enables the training of markedly overparameterized GCN models. Remarkably, with GIST, we train an astonishgly-wide 32,768-dimensional GraphSAGE model, which exceeds the capacity of a single GPU by a factor of 8x, to SOTA performance on the Amazon2M dataset.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 35 of its 59 references that are in the map. First posted on arXiv 20 Feb 2021.

Builds on

35 of 59 references.

References · 35

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  5. 2017Graph Attention Networkscited by 8,854
  6. 2017Inductive Representation Learning on Large Graphscited by 6,284
  7. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  8. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  9. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,148
  10. 2016Convolutional Neural Networks on Graphs with Fast Localized Spectral Filteringcited by 2,085
  11. 2018Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learningcited by 1,099
  12. 2016Geometric deep learning: going beyond Euclidean datacited by 1,042
23 more
  1. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  2. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  3. 2019GraphSAINT: Graph Sampling Based Inductive Learning Methodcited by 618
  4. 2018Local SGD Converges Fast and Communicates Littlecited by 595
  5. 2018FastGCN: Fast Learning with Graph Convolutional Networks via Importance Samplingcited by 583
  6. 2019Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networkscited by 540
  7. 2017Can Decentralized Algorithms Outperform Centralized Algorithms? A Case Study for Decentralized Parallel Stochastic Gradient Descentcited by 496
  8. 2018Don't Use Large Mini-Batches, Use Local SGDcited by 201
  9. 2018Adaptive Sampling Towards Fast Graph Representation Learningcited by 180
  10. 2019Layer-Dependent Importance Sampling for Training Deep and Large Graph Convolutional Networkscited by 177
  11. 2018Demystifying Parallel and Distributed Deep Learning: An In-Depth Concurrency Analysiscited by 154
  12. 2018Large-Scale Learnable Graph Convolutional Networkscited by 113
  13. 2020The Cost of Training NLP Models: A Concise Overviewcited by 97
  14. 2014Deep learning with Elastic Averaging SGDcited by 90
  15. 2019Quadratic Suffices for Over-parametrization via Matrix Chernoff Boundcited by 58
  16. 2020L2-GCN: Layer-Wise and Learned Efficient Training of Graph Convolutional Networkscited by 48
  17. 2018Layer-Parallel Training of Deep Residual Neural Networkscited by 35
  18. 2020Convolutional Neural Network Training with Distributed K-FACcited by 22
  19. 2017Integrated Model, Batch and Domain Parallelism in Training Neural Networkscited by 16
  20. 2020LAMP: Large Deep Nets with Automated Model Parallelism for Image Segmentationcited by 8
  21. 2020Layer-Parallel Training with GPU Concurrency of Deep Residual Neural Networks via Nonlinear Multigridcited by 8
  22. 2019Layered SGD: A Decentralized and Synchronous SGD Algorithm for Scalable Deep Neural Network Trainingcited by 2
  23. 2019Automatic Model Parallelism for Deep Neural Networks with Compiler and Hardware Supportcited by 2

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 3; 2023: 3; 2024: 1; 2025: 1.

10 papers.

  1. 2022Distributed Graph Neural Network Training: A Surveycited by 32
  2. 2024FedP3: Federated Personalized and Privacy-friendly Network Pruning under Model Heterogeneitycited by 13
  3. 2022Efficient and Light-Weight Federated Learning via Asynchronous Distributed Dropoutcited by 12
  4. 2021On the Convergence of Shallow Neural Network Training with Randomly Masked Neuronscited by 11
  5. 2022SUGAR: Efficient Subgraph-level Training via Resource-aware Graph Partitioningcited by 6
  6. 2023Towards a Better Theoretical Understanding of Independent Subnetwork Trainingcited by 6
  7. 2023Federated Learning Over Images: Vertical Decompositions and Pre-Trained Backbones Are Difficult to Beatcited by 4
  8. 2021Label Propagation across Graphs: Node Classification using Graph Neural Tangent Kernelscited by 2
  9. 2023Federated Learning over Hierarchical Wireless Networks: Training Latency Minimization via Submodel Partitioningcited by 1
  10. 2025Neighbor-Sampling Based Momentum Stochastic Methods for Training Graph Neural Networkscited by 0