mlsanityData to 14 Oct 2026

A Comprehensive Study on Large-Scale Graph Training: Benchmarking and Rethinking

arXiv 2210.07494 ↗First posted 14 Oct 2022cs.LG cs.AIcited by 56

Not on the map. Why?

Abstract (arXiv)

Large-scale graph training is a notoriously challenging problem for graph neural networks (GNNs). Due to the nature of evolving graph structures into the training process, vanilla GNNs usually fail to scale up, limited by the GPU memory space. Up to now, though numerous scalable GNN architectures have been proposed, we still lack a comprehensive survey and fair benchmark of this reservoir to find the rationale for designing scalable GNNs. To this end, we first systematically formulate the representative methods of large-scale graph training into several branches and further establish a fair and consistent benchmark for them by a greedy hyperparameter searching. In addition, regarding efficiency, we theoretically evaluate the time and space complexity of various branches and empirically compare them w.r.t GPU memory usage, throughput, and convergence. Furthermore, We analyze the pros and cons for various branches of scalable GNNs and then present a new ensembling training manner, named EnGCN, to address the existing issues. Our code is available at https://github.com/VITA-Group/Large_Scale_GCN_Benchmarking.

Key facts. Cited by 56 papers in the map (data through 14 Oct 2026). Builds on 45 of its 66 references that are in the map. First posted on arXiv 14 Oct 2022.

Builds on

45 of 66 references.

References · 45

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  3. 2017Graph Attention Networkscited by 8,923
  4. 2017Inductive Representation Learning on Large Graphscited by 6,320
  5. 2018How Powerful are Graph Neural Networks?cited by 4,393
  6. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  7. 2019Fast Graph Representation Learning with PyTorch Geometriccited by 2,434
  8. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,223
  9. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,163
  10. 2019Simplifying Graph Convolutional Networkscited by 1,477
  11. 2020Graph Contrastive Learning with Augmentationscited by 1,208
  12. 2020LightGCN: Simplifying and Powering Graph Convolution Network for Recommendationcited by 1,139
33 more
  1. 2018Graph Convolutional Neural Networks for Web-Scale Recommender Systemscited by 1,138
  2. 2018Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learningcited by 1,102
  3. 2018Predict then Propagate: Graph Neural Networks meet Personalized PageRankcited by 993
  4. 2019Strategies for Pre-training Graph Neural Networkscited by 971
  5. 2018Representation Learning on Graphs with Jumping Knowledge Networkscited by 877
  6. 2019GraphSAINT: Graph Sampling Based Inductive Learning Methodcited by 618
  7. 2018FastGCN: Fast Learning with Graph Convolutional Networks via Importance Samplingcited by 584
  8. 2019Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networkscited by 541
  9. 2019DeepGCNs: Can GCNs Go as Deep as CNNs?cited by 527
  10. 2019Graph U-Netscited by 492
  11. 2020SIGN: Scalable Inception Graph Neural Networkscited by 284
  12. 2020DeeperGCN: All You Need to Train Deeper GCNscited by 250
  13. 2018Learning to Propagate Labels: Transductive Propagation Network for Few-shot Learningcited by 211
  14. 2020Combining Label Propagation and Simple Models Out-performs Graph Neural Networkscited by 202
  15. 2018Adaptive Sampling Towards Fast Graph Representation Learningcited by 180
  16. 2019Layer-Dependent Importance Sampling for Training Deep and Large Graph Convolutional Networkscited by 177
  17. 2020Scaling Graph Neural Networks with Approximate PageRankcited by 174
  18. 2017Predicting multicellular function through multi-layer tissue networkscited by 138
  19. 2021Node Feature Extraction by Self-Supervised Multi-scale Neighborhood Predictioncited by 138
  20. 2020Towards Deeper Graph Neural Networks with Differentiable Group Normalizationcited by 114
  21. 2018Large-Scale Learnable Graph Convolutional Networkscited by 113
  22. 2020Unifying Graph Convolutional Neural Networks and Label Propagationcited by 98
  23. 2021Dirichlet Energy Constrained Learning for Deep Graph Neural Networkscited by 97
  24. 2021GNNAutoScale: Scalable and Expressive Graph Neural Networks via Historical Embeddingscited by 90
  25. 2022Graph Attention Multi-Layer Perceptroncited by 83
  26. 2021ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Trainingcited by 61
  27. 2021DistGNN: Scalable Distributed Training for Large-Scale Graph Neural Networkscited by 59
  28. 2021Cold Brew: Distilling Graph Node Representations with Incomplete or Missing Neighborhoodscited by 58
  29. 2020Minimal Variance Sampling with Provable Guarantees for Fast Training of Graph Neural Networkscited by 54
  30. 2021VQ-GNN: A Universal Framework to Scale up Graph Neural Networks using Vector Quantizationcited by 39
  31. 2019AdaGCN: Adaboosting Graph Convolutional Networks into Deep Modelscited by 37
  32. 2021Bag of Tricks for Training Deeper Graph Neural Networks: A Comprehensive Benchmark Studycited by 36
  33. 2022Neighbor2Seq: Deep Learning on Massive Graphs by Transforming Neighbors to Sequencescited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 2; 2023: 19; 2024: 21; 2025: 10; 2026: 4.

56 papers.

Sort:

  1. 2023SimTeG: A Frustratingly Simple Approach Improves Textual Graph Learningcited by 73
  2. 2023Cluster-guided Contrastive Graph Clustering Networkcited by 60
  3. 2024Foundation Models for the Electric Power Gridcited by 28
  4. 2024IDEA: A Flexible Framework of Certified Unlearning for Graph Neural Networkscited by 22
  5. 2024EXGC: Bridging Efficiency and Explainability in Graph Condensationcited by 20
  6. 2024Design2GarmentCode: Turning Design Concepts to Tangible Garments Through Program Synthesiscited by 18
  7. 2023LazyGNN: Large-Scale Graph Neural Networks via Lazy Propagationcited by 16
  8. 2023Graph Neural Networks for Road Safety Modeling: Datasets and Evaluations for Accident Analysiscited by 13
  9. 2022RSC: Accelerating Graph Neural Networks Training via Randomized Sparse Computationscited by 12
  10. 2023Federated Learning of Medical Concepts Embedding using BEHRTcited by 11
  11. 2023Attend Who is Weak: Enhancing Graph Condensation via Cross-Free Adversarial Trainingcited by 9
  12. 2024A Survey on Graph Condensationcited by 9
38 more
  1. 2024Two Trades is not Baffled: Condensing Graph via Crafting Rational Gradient Matchingcited by 8
  2. 2025GiGL: Large-Scale Graph Neural Networks at Snapchatcited by 8
  3. 2023HiGen: Hierarchical Graph Generative Networkscited by 6
  4. 2023Efficient GNN Explanation via Learning Removal-based Attributioncited by 6
  5. 2023A benchmark of categorical encoders for binary classificationcited by 6
  6. 2024A Survey of Few-Shot Learning on Graphs: from Meta-Learning to Pre-Training and Prompt Learningcited by 6
  7. 2024A Comprehensive Benchmark on Spectral GNNs: The Impact on Efficiency, Memory, and Effectivenesscited by 6
  8. 2024Rethinking Independent Cross-Entropy Loss For Graph-Structured Datacited by 5
  9. 2022From Node Interaction to Hop Interaction: New Effective and Scalable Graph Learning Paradigmcited by 4
  10. 2023A Comprehensive Survey on Graph Summarization with Graph Neural Networkscited by 4
  11. 2023You Only Transfer What You Share: Intersection-Induced Graph Transfer Learning for Link Predictioncited by 4
  12. 2023Bkd-FedGNN: A Benchmark for Classification Backdoor Attacks on Federated Graph Neural Networkcited by 4
  13. 2023Activation Compression of Graph Neural Networks using Block-wise Quantization with Improved Variance Minimizationcited by 4
  14. 2023When Graph Neural Network Meets Causality: Opportunities, Methodologies and An Outlookcited by 4
  15. 2024Ada-HGNN: Adaptive Sampling for Scalable Hypergraph Neural Networkscited by 4
  16. 2024GraphFM: A Comprehensive Benchmark for Graph Foundation Modelcited by 4
  17. 2024Rethinking the Effectiveness of Graph Classification Datasets in Benchmarks for Assessing GNNscited by 4
  18. 2025Efficient Graph Condensation via Gaussian Processcited by 4
  19. 2023Editable Graph Neural Network for Node Classificationscited by 3
  20. 2024STAGE: Simplified Text-Attributed Graph Embeddings Using Pre-trained LLMscited by 3
  21. 2024All Against Some: Efficient Integration of Large Language Models for Message Passing in Graph Neural Networkscited by 3
  22. 2024Beyond Grid Data: Exploring Graph Neural Networks for Earth Observationcited by 3
  23. 2026CITED: A Decision Boundary-Aware Signature for GNNs Towards Model Extraction Defensecited by 3
  24. 2023Knowledge Enhanced Graph Neural Networks for Graph Completioncited by 2
  25. 2023A Simple and Scalable Graph Neural Network for Large Directed Graphscited by 2
  26. 2023Graph Ladling: Shockingly Simple Parallel GNN Training without Intermediate Communicationcited by 2
  27. 2025GBGC: Efficient and Adaptive Graph Coarsening via Granular-ball Computingcited by 2
  28. 2025LPS-GNN : Deploying Graph Neural Networks on Graphs with 100-Billion Edgescited by 2
  29. 2025Scaling Kinetic Monte-Carlo Simulations of Grain Growth with Combined Convolutional and Graph Neural Networkscited by 2
  30. 2024On the Robustness of Graph Reduction Against GNN Backdoorcited by 1
  31. 2024Benchmark on Drug Target Interaction Modeling from a Drug Structure Perspectivecited by 1
  32. 2024Gradient Rewiring for Editable Graph Neural Network Trainingcited by 1
  33. 2024Disentangled and Self-Explainable Node Representation Learningcited by 1
  34. 2025Toward Efficient Testing of Graph Neural Networks via Test Input Prioritizationcited by 1
  35. 2023Entropy Aware Training for Fast and Accurate Distributed GNNcited by 0
  36. 2024Sparse Decomposition of Graph Neural Networkscited by 0
  37. 2024ReHub: Linear Complexity Graph Transformers with Adaptive Hub-Spoke Reassignmentcited by 0
  38. 2025Theoretical Learning Performance of Graph Neural Networks: The Impact of Jumping Connections and Layer-wise Sparsificationcited by 0

50 most cited of 56 All 56 (JSON)