mlsanityData to 14 Oct 2026

Training Graph Neural Networks with 1000 Layers

arXiv 2106.07476 ↗First posted 14 Jun 2021cs.LG cs.AI cs.SIcited by 169

Not on the map. Why?

Abstract (arXiv)

Deep graph neural networks (GNNs) have achieved excellent results on various tasks on increasingly large graph datasets with millions of nodes and edges. However, memory complexity has become a major obstacle when training deep GNNs for practical applications due to the immense number of nodes, edges, and intermediate activations. To improve the scalability of GNNs, prior works propose smart graph sampling or partitioning strategies to train GNNs with a smaller set of nodes or sub-graphs. In this work, we study reversible connections, group convolutions, weight tying, and equilibrium models to advance the memory and parameter efficiency of GNNs. We find that reversible connections in combination with deep network architectures enable the training of overparameterized GNNs that significantly outperform existing methods on multiple datasets. Our models RevGNN-Deep (1001 layers with 80 channels each) and RevGNN-Wide (448 layers with 224 channels each) were both trained on a single commodity GPU and achieve an ROC-AUC of 87.74 ± 0.13 and 88.24 ± 0.15 on the ogbn-proteins dataset. To the best of our knowledge, RevGNN-Deep is the deepest GNN in the literature by one order of magnitude. Please visit our project website https://www.deepgcns.org/arch/gnn1000 for more information.

Key facts. Cited by 169 papers in the map (data through 14 Oct 2026). Builds on 63 of its 78 references that are in the map. First posted on arXiv 14 Jun 2021.

Builds on

63 of 78 references.

References · 63

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  5. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  6. 2017Graph Attention Networkscited by 8,923
  7. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  8. 2016Densely Connected Convolutional Networkscited by 6,720
  9. 2017Inductive Representation Learning on Large Graphscited by 6,320
  10. 2018How Powerful are Graph Neural Networks?cited by 4,393
  11. 2017Neural Message Passing for Quantum Chemistrycited by 3,487
  12. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,546
38 more
  1. 2019Fast Graph Representation Learning with PyTorch Geometriccited by 2,434
  2. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,223
  3. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,163
  4. 2016Convolutional Neural Networks on Graphs with Fast Localized Spectral Filteringcited by 2,089
  5. 2020Reformer: The Efficient Transformercited by 1,668
  6. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  7. 2019Simplifying Graph Convolutional Networkscited by 1,477
  8. 2015Multi-Scale Context Aggregation by Dilated Convolutionscited by 1,133
  9. 2013Spectral Networks and Locally Connected Networks on Graphscited by 1,102
  10. 2018Deeper Insights into Graph Convolutional Networks for Semi-Supervised Learningcited by 1,102
  11. 2018Pitfalls of Graph Neural Network Evaluationcited by 1,007
  12. 2018Predict then Propagate: Graph Neural Networks meet Personalized PageRankcited by 993
  13. 2016Revisiting Semi-Supervised Learning with Graph Embeddingscited by 976
  14. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  15. 2018Representation Learning on Graphs with Jumping Knowledge Networkscited by 877
  16. 2020Simple and Deep Graph Convolutional Networkscited by 860
  17. 2019DropEdge: Towards Deep Graph Convolutional Networks on Node Classificationcited by 680
  18. 2020Benchmarking Graph Neural Networkscited by 680
  19. 2019GraphSAINT: Graph Sampling Based Inductive Learning Methodcited by 618
  20. 2019Deep Equilibrium Modelscited by 615
  21. 2018FastGCN: Fast Learning with Graph Convolutional Networks via Importance Samplingcited by 584
  22. 2019Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networkscited by 541
  23. 2019DeepGCNs: Can GCNs Go as Deep as CNNs?cited by 527
  24. 2020Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classificationcited by 506
  25. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 497
  26. 2016Geometric deep learning on graphs and manifolds using mixture model CNNscited by 468
  27. 2021OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphscited by 329
  28. 2016Learning Convolutional Neural Networks for Graphscited by 314
  29. 2019PairNorm: Tackling Oversmoothing in GNNscited by 297
  30. 2020Towards Deeper Graph Neural Networkscited by 296
  31. 2020SIGN: Scalable Inception Graph Neural Networkscited by 284
  32. 2016Using the Output Embedding to Improve Language Modelscited by 272
  33. 2020DeeperGCN: All You Need to Train Deeper GCNscited by 250
  34. 2015Deep Convolutional Networks on Graph-Structured Datacited by 246
  35. 2017The Reversible Residual Network: Backpropagation Without Storing Activationscited by 217
  36. 2017CayleyNets: Graph Convolutional Neural Networks with Complex Rational Spectral Filterscited by 203
  37. 2020Scaling Graph Neural Networks with Approximate PageRankcited by 174
  38. 2018Adaptive Graph Convolutional Neural Networkscited by 155

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 17; 2022: 37; 2023: 46; 2024: 39; 2025: 20; 2026: 10.

169 papers.

Sort:

By area:

  1. 2022PointNeXt: Revisiting PointNet++ with Improved Training and Scaling Strategiescited by 436
  2. 2023Exploring the Potential of Large Language Models (LLMs) in Learning on Graphscited by 274
  3. 2023GraphGPT: Graph Instruction Tuning for Large Language Modelscited by 226
  4. 2023Language is All a Graph Needscited by 220
  5. 2023On Over-Squashing in Message Passing Neural Networks: The Impact of Width, Depth, and Topologycited by 166
  6. 2022Learning on Large-scale Text-attributed Graphs via Variational Inferencecited by 155
  7. 2021Graph-less Neural Networks: Teaching Old MLPs New Tricks via Distillationcited by 152
  8. 2021Node Feature Extraction by Self-Supervised Multi-scale Neighborhood Predictioncited by 138
  9. 2023Harnessing Explanations: LLM-to-LM Interpreter for Enhanced Text-Attributed Graph Representation Learningcited by 133
  10. 2022Trustworthy Graph Neural Networks: Aspects, Methods and Trendscited by 89
  11. 2022BOND: Benchmarking Unsupervised Outlier Node Detection on Static Attributed Graphscited by 80
  12. 2023SimTeG: A Frustratingly Simple Approach Improves Textual Graph Learningcited by 73
38 more
  1. 2022Revisiting Over-smoothing in BERT from the Perspective of Graphcited by 72
  2. 2024Can GNN be Good Adapter for LLMs?cited by 71
  3. 2022A Survey on Graph Representation Learning Methodscited by 66
  4. 2022Ordered Subgraph Aggregation Networkscited by 57
  5. 2021Simplifying approach to Node Classification in Graph Neural Networkscited by 56
  6. 2022A Survey on Graph Neural Networks and Graph Transformers in Computer Vision: A Task-Oriented Perspectivecited by 56
  7. 2021Local Augmentation for Graph Neural Networkscited by 53
  8. 2024AnyGraph: Graph Foundation Model in the Wildcited by 53
  9. 2021Simple GNN Regularisation for 3D Molecular Property Prediction & Beyondcited by 44
  10. 2023ContraNorm: A Contrastive Learning Perspective on Oversmoothing and Beyondcited by 39
  11. 2021On Representation Knowledge Distillation for Graph Neural Networkscited by 38
  12. 2023Dink-Net: Neural Clustering on Large Graphscited by 38
  13. 2021Bag of Tricks for Training Deeper Graph Neural Networks: A Comprehensive Benchmark Studycited by 36
  14. 2023Reversible Vision Transformerscited by 33
  15. 2022GraphFM: Improving Large-Scale GNN Training via Feature Momentumcited by 31
  16. 2022MLPInit: Embarrassingly Simple GNN Training Acceleration with MLP Initializationcited by 31
  17. 2022Parallel and Distributed Graph Neural Networks: An In-Depth Concurrency Analysiscited by 29
  18. 2023Train Your Own GNN Teacher: Graph-Aware Distillation on Textual Graphscited by 29
  19. 2024ProG: A Graph Prompt Learning Benchmarkcited by 29
  20. 2023GRENADE: Graph-Centric Language Model for Self-Supervised Representation Learning on Text-Attributed Graphscited by 28
  21. 2021Scalable deeper graph neural networks for high-performance materials property predictioncited by 26
  22. 2023Towards Deep Attention in Graph Neural Networks: Problems and Remediescited by 24
  23. 2024When Graph meets Multimodal: Benchmarking and Meditating on Multimodal Attributed Graphs Learningcited by 22
  24. 2025LLM as GNN: Graph Vocabulary Learning for Text-Attributed Graph Foundation Modelscited by 22
  25. 2025GraphICL: Unlocking Graph Learning Potential in LLMs through Structured Prompt Designcited by 21
  26. 2024Graph-oriented Instruction Tuning of Large Language Models for Generic Graph Miningcited by 20
  27. 2021Bridging the Gap between Spatial and Spectral Domains: A Unified Framework for Graph Neural Networkscited by 19
  28. 2021Evaluating Deep Graph Neural Networkscited by 19
  29. 2022Label-invariant Augmentation for Semi-Supervised Graph Classificationcited by 19
  30. 2022Feature Overcorrelation in Deep Graph Neural Networks: A New Perspectivecited by 19
  31. 2022MGNNI: Multiscale Graph Neural Networks with Implicit Layerscited by 19
  32. 2022Re^2TAL: Rewiring Pretrained Video Backbones for Reversible Temporal Action Localizationcited by 19
  33. 2021Sequential Aggregation and Rematerialization: Distributed Full-batch Training of Graph Neural Networks on Large Graphscited by 18
  34. 2022Compressing Deep Graph Neural Networks via Adversarial Knowledge Distillationcited by 18
  35. 2023A Survey on Graph Neural Network Acceleration: Algorithms, Systems, and Customized Hardwarecited by 18
  36. 2023Rank Collapse Causes Over-Smoothing and Over-Correlation in Graph Neural Networkscited by 18
  37. 2024Graph Sparsification via Mixture of Graphscited by 17
  38. 2023LazyGNN: Large-Scale Graph Neural Networks via Lazy Propagationcited by 16

50 most cited of 169 All 169 (JSON)