mlsanityData to 7 Oct 2026

GNNAutoScale: Scalable and Expressive Graph Neural Networks via Historical Embeddings

arXiv 2106.05609 ↗First posted 10 Jun 2021cs.LGcited by 90

Not on the map. Why?

Abstract (arXiv)

We present GNNAutoScale (GAS), a framework for scaling arbitrary message-passing GNNs to large graphs. GAS prunes entire sub-trees of the computation graph by utilizing historical embeddings from prior training iterations, leading to constant GPU memory consumption in respect to input node size without dropping any data. While existing solutions weaken the expressive power of message passing due to sub-sampling of edges or non-trainable propagations, our approach is provably able to maintain the expressive power of the original GNN. We achieve this by providing approximation error bounds of historical embeddings and show how to tighten them in practice. Empirically, we show that the practical realization of our framework, PyGAS, an easy-to-use extension for PyTorch Geometric, is both fast and memory-efficient, learns expressive node representations, closely resembles the performance of their non-scaling counterparts, and reaches state-of-the-art performance on large-scale graphs.

Key facts. Cited by 90 papers in the map (data through 7 Oct 2026). Builds on 42 of its 63 references that are in the map. First posted on arXiv 10 Jun 2021.

Builds on

42 of 63 references.

References · 42

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  3. 2017Graph Attention Networkscited by 8,854
  4. 2017Inductive Representation Learning on Large Graphscited by 6,284
  5. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,384
  6. 2018How Powerful are Graph Neural Networks?cited by 4,355
  7. 2017Neural Message Passing for Quantum Chemistrycited by 3,460
  8. 2019Fast Graph Representation Learning with PyTorch Geometriccited by 2,414
  9. 2018Dynamic Graph CNN for Learning on Point Cloudscited by 2,220
  10. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,148
  11. 2019Simplifying Graph Convolutional Networkscited by 1,470
  12. 2017Deep Setscited by 1,394
30 more
  1. 2018Pitfalls of Graph Neural Network Evaluationcited by 994
  2. 2018Predict then Propagate: Graph Neural Networks meet Personalized PageRankcited by 987
  3. 2016Revisiting Semi-Supervised Learning with Graph Embeddingscited by 970
  4. 2018Weisfeiler and Leman Go Neural: Higher-order Graph Neural Networkscited by 878
  5. 2018Representation Learning on Graphs with Jumping Knowledge Networkscited by 875
  6. 2020Simple and Deep Graph Convolutional Networkscited by 857
  7. 2019DropEdge: Towards Deep Graph Convolutional Networks on Node Classificationcited by 679
  8. 2020Benchmarking Graph Neural Networkscited by 672
  9. 2019GraphSAINT: Graph Sampling Based Inductive Learning Methodcited by 618
  10. 2018FastGCN: Fast Learning with Graph Convolutional Networks via Importance Samplingcited by 583
  11. 2019Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networkscited by 540
  12. 2020Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classificationcited by 500
  13. 2020Principal Neighbourhood Aggregation for Graph Netscited by 477
  14. 2019Diffusion Improves Graph Learningcited by 414
  15. 2019Provably Powerful Graph Networkscited by 385
  16. 2020SIGN: Scalable Inception Graph Neural Networkscited by 283
  17. 2020Wiki-CS: A Wikipedia-Based Benchmark for Graph Neural Networkscited by 253
  18. 2020Combining Label Propagation and Simple Models Out-performs Graph Neural Networkscited by 199
  19. 2018Adaptive Sampling Towards Fast Graph Representation Learningcited by 180
  20. 2019Layer-Dependent Importance Sampling for Training Deep and Large Graph Convolutional Networkscited by 177
  21. 2019AliGraph: A Comprehensive Graph Neural Network Platformcited by 152
  22. 2020DistDGL: Distributed Graph Neural Network Training for Billion-Scale Graphscited by 139
  23. 2020Scalable Graph Neural Networks via Bidirectional Propagationcited by 87
  24. 2020Minimal Variance Sampling with Provable Guarantees for Fast Training of Graph Neural Networkscited by 54
  25. 2020Scalable Graph Neural Networks for Heterogeneous Graphscited by 31
  26. 2020Deep Graph Neural Networks with Shallow Subgraph Samplerscited by 16
  27. 2021Graph Traversal with Tensor Functionals: A Meta-Algorithm for Scalable Learningcited by 12
  28. 2020Distributed Training of Graph Convolutional Networks using Subgraph Approximationcited by 7
  29. 2018Towards Robust Neural Networks with Lipschitz Continuitycited by 6
  30. 2019Discriminative structural graph classificationcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 25; 2023: 30; 2024: 15; 2025: 10; 2026: 3.

90 papers.

Sort:

  1. 2023SGFormer: Simplifying and Empowering Transformers for Large-Graph Representationscited by 131
  2. 2022Trustworthy Graph Neural Networks: Aspects, Methods and Trendscited by 88
  3. 2024Position: Graph Foundation Models are Already Herecited by 84
  4. 2022A Survey on Graph Representation Learning Methodscited by 66
  5. 2022Ordered Subgraph Aggregation Networkscited by 57
  6. 2022A Comprehensive Study on Large-Scale Graph Training: Benchmarking and Rethinkingcited by 56
  7. 2025PyG 2.0: Scalable Learning on Real World Graphscited by 49
  8. 2022BNS-GCN: Efficient Full-Graph Training of Graph Convolutional Networks with Partition-Parallelism and Random Boundary Node Samplingcited by 43
  9. 2022Linkless Link Prediction via Relational Distillationcited by 39
  10. 2023VQGraph: Rethinking Graph Representation Space for Bridging GNNs and MLPscited by 34
  11. 2022GraphFM: Improving Large-Scale GNN Training via Feature Momentumcited by 31
  12. 2022MLPInit: Embarrassingly Simple GNN Training Acceleration with MLP Initializationcited by 31
38 more
  1. 2023LMC: Fast Training of GNNs via Subgraph Sampling with Provable Convergencecited by 26
  2. 2021SSSNET: Semi-Supervised Signed Network Clusteringcited by 20
  3. 2022SCARA: Scalable Graph Neural Networks with Feature-Oriented Optimizationcited by 19
  4. 2022Layer-Neighbor Sampling -- Defusing Neighborhood Explosion in GNNscited by 19
  5. 2021Bridging the Gap between Spatial and Spectral Domains: A Unified Framework for Graph Neural Networkscited by 18
  6. 2022Compressing Deep Graph Neural Networks via Adversarial Knowledge Distillationcited by 18
  7. 2023A Survey on Graph Neural Network Acceleration: Algorithms, Systems, and Customized Hardwarecited by 18
  8. 2024Anomaly Detection in Dynamic Graphs: A Comprehensive Surveycited by 18
  9. 2021Bag-of-Words vs. Graph vs. Sequence in Text Classification: Questioning the Necessity of Text-Graphs and the Surprising Strength of a Wide MLPcited by 17
  10. 2021SMORE: Knowledge Graph Completion and Multi-hop Reasoning in Massive Knowledge Graphscited by 17
  11. 2023Hierarchical Transformer for Scalable Graph Learningcited by 17
  12. 2023LazyGNN: Large-Scale Graph Neural Networks via Lazy Propagationcited by 16
  13. 2021SelfCF: A Simple Framework for Self-supervised Collaborative Filteringcited by 15
  14. 2022Influence-Based Mini-Batching for Graph Neural Networkscited by 15
  15. 2024Sketch-GNN: Scalable Graph Neural Networks with Sublinear Training Complexitycited by 14
  16. 2023The Evolution of Distributed Systems for Graph Neural Networks and their Origin in Graph Processing and Deep Learning: A Surveycited by 13
  17. 2024Graph Neural Networks for Tabular Data Learning: A Survey with Taxonomy and Directionscited by 13
  18. 2021Implicit vs Unfolded Graph Neural Networkscited by 12
  19. 2022ReFactor GNNs: Revisiting Factorisation-based Models from a Message-Passing Perspectivecited by 12
  20. 2022RSC: Accelerating Graph Neural Networks Training via Randomized Sparse Computationscited by 12
  21. 2023ProGAP: Progressive Graph Neural Networks with Differential Privacy Guaranteescited by 12
  22. 2025DeepGate4: Efficient and Effective Representation Learning for Circuit Design at Scalecited by 12
  23. 2022Taxonomy of Benchmarks in Graph Representation Learningcited by 11
  24. 2023Otter-Knowledge: benchmarks of multimodal knowledge graph representation learning from different sources for drug discoverycited by 10
  25. 2022Sampling Enclosing Subgraphs for Link Predictioncited by 9
  26. 2023Learning Large Graph Property Prediction via Graph Segment Trainingcited by 9
  27. 2021IGLU: Efficient GCN Training via Lazy Updatescited by 7
  28. 2023Simplifying Distributed Neural Network Training on Massive Graphs: Randomized Partitions Improve Model Aggregationcited by 7
  29. 2023Label Deconvolution for Node Representation Learning on Large-scale Attributed Graphs against Learning Biascited by 7
  30. 2023GRAPES: Learning to Sample Graphs for Scalable Graph Neural Networkscited by 7
  31. 2023Mixture of Weak & Strong Experts on Graphscited by 7
  32. 2022Graph Generative Model for Benchmarking Graph Neural Networkscited by 6
  33. 2023Staleness-Alleviated Distributed GNN Training via Online Dynamic-Embedding Predictioncited by 6
  34. 2022Learning Stochastic Graph Neural Networks with Constrained Variancecited by 5
  35. 2023NeutronOrch: Rethinking Sample-based GNN Training under CPU-GPU Heterogeneous Environmentscited by 5
  36. 2022BiFeat: Supercharge GNN Training via Graph Feature Quantizationcited by 4
  37. 2023FreshGNN: Reducing Memory Access via Stable Historical Embeddings for Graph Neural Network Trainingcited by 4
  38. 2023Tango: rethinking quantization for graph neural network training on GPUscited by 4

50 most cited of 90 All 90 (JSON)