mlsanityData to 14 Oct 2026

Sequential Aggregation and Rematerialization: Distributed Full-batch Training of Graph Neural Networks on Large Graphs

arXiv 2111.06483 ↗First posted 11 Nov 2021cs.LG cs.AIcited by 18

Not on the map. Why?

Abstract (arXiv)

We present the Sequential Aggregation and Rematerialization (SAR) scheme for distributed full-batch training of Graph Neural Networks (GNNs) on large graphs. Large-scale training of GNNs has recently been dominated by sampling-based methods and methods based on non-learnable message passing. SAR on the other hand is a distributed technique that can train any GNN type directly on an entire large graph. The key innovation in SAR is the distributed sequential rematerialization scheme which sequentially re-constructs then frees pieces of the prohibitively large GNN computational graph during the backward pass. This results in excellent memory scaling behavior where the memory consumption per worker goes down linearly with the number of workers, even for densely connected graphs. Using SAR, we report the largest applications of full-batch GNN training to-date, and demonstrate large memory savings as the number of workers increases. We also present a general technique based on kernel fusion and attention-matrix rematerialization to optimize both the runtime and memory efficiency of attention-based models. We show that, coupled with SAR, our optimized attention kernels lead to significant speedups and memory savings in attention-based GNNs.We made the SAR GNN training library publicy available: urlhttps://github.com/IntelLabs/SAR.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Builds on 32 of its 44 references that are in the map. First posted on arXiv 11 Nov 2021.

Builds on

32 of 44 references.

References · 32

  1. 2017Attention Is All You Needcited by 60,361
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  4. 2017Graph Attention Networkscited by 8,923
  5. 2017Inductive Representation Learning on Large Graphscited by 6,320
  6. 2017Neural Message Passing for Quantum Chemistrycited by 3,487
  7. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,163
  8. 2017Modeling Relational Data with Graph Convolutional Networkscited by 1,694
  9. 2019Simplifying Graph Convolutional Networkscited by 1,477
  10. 2018Predict then Propagate: Graph Neural Networks meet Personalized PageRankcited by 993
  11. 2018Representation Learning on Graphs with Jumping Knowledge Networkscited by 877
  12. 2016Training Deep Nets with Sublinear Memory Costcited by 848
20 more
  1. 2018Link Prediction Based on Graph Neural Networkscited by 832
  2. 2019GraphSAINT: Graph Sampling Based Inductive Learning Methodcited by 618
  3. 2018FastGCN: Fast Learning with Graph Convolutional Networks via Importance Samplingcited by 584
  4. 2019Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networkscited by 541
  5. 2020Masked Label Prediction: Unified Message Passing Model for Semi-Supervised Classificationcited by 506
  6. 2020Principal Neighbourhood Aggregation for Graph Netscited by 478
  7. 2020SIGN: Scalable Inception Graph Neural Networkscited by 284
  8. 2020DeeperGCN: All You Need to Train Deeper GCNscited by 250
  9. 2019Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networkscited by 242
  10. 2017The Reversible Residual Network: Backpropagation Without Storing Activationscited by 217
  11. 2019Graph Matching Networks for Learning the Similarity of Graph Structured Objectscited by 205
  12. 2020Combining Label Propagation and Simple Models Out-performs Graph Neural Networkscited by 202
  13. 2021Training Graph Neural Networks with 1000 Layerscited by 169
  14. 2020DistDGL: Distributed Graph Neural Network Training for Billion-Scale Graphscited by 139
  15. 2019Checkmate: Breaking the Memory Wall with Optimal Tensor Rematerializationcited by 91
  16. 2021DistGNN: Scalable Distributed Training for Large-Scale Graph Neural Networkscited by 59
  17. 2021Scalable and Adaptive Graph Neural Networks with Self-Label-Enhanced trainingcited by 54
  18. 2020Reducing Communication in Graph Neural Network Trainingcited by 50
  19. 2017Integrated Model, Batch and Domain Parallelism in Training Neural Networkscited by 16
  20. 2019Improving Strong-Scaling of CNN Training by Exploiting Finer-Grained Parallelismcited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 4; 2023: 6; 2024: 2; 2025: 3; 2026: 3.

18 papers.

Sort:

By area:

  1. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 258
  2. 2022Distributed Graph Neural Network Training: A Surveycited by 32
  3. 2022Parallel and Distributed Graph Neural Networks: An In-Depth Concurrency Analysiscited by 29
  4. 2023A Survey on Graph Neural Network Acceleration: Algorithms, Systems, and Customized Hardwarecited by 18
  5. 2022A Comprehensive Survey on Distributed Training of Graph Neural Networkscited by 15
  6. 2022The Open MatSci ML Toolkit: A Flexible Framework for Machine Learning in Materials Sciencecited by 14
  7. 2024Graph Neural Network Training Systems: A Performance Comparison of Full-Graph and Mini-Batchcited by 10
  8. 2023A Graph Neural Network with Negative Message Passing for Graph Coloringcited by 5
  9. 2023GNNPipe: Scaling Deep GNN Training with Pipelined Model Parallelismcited by 4
  10. 2023Boosting Distributed Full-graph GNN Training with Asynchronous One-bit Communicationcited by 3
  11. 2023Communication-Free Distributed GNN Training with Vertex Cutcited by 2
  12. 2025HydroGAT: Distributed Heterogeneous Graph Attention Transformer for Spatiotemporal Flood Predictioncited by 1
6 more
  1. 2023Moccasin: Efficient Tensor Rematerialization for Neural Networkscited by 0
  2. 2024Distributed Training of Large Graph Neural Networks with Variable Communication Ratescited by 0
  3. 2025Distributed Equivariant Graph Neural Networks for Large-Scale Electronic Structure Predictioncited by 0
  4. 2026Not All Neighbors Matter: Understanding the Impact of Graph Sparsification on GNN Pipelinescited by 0
  5. 2026SNI-GNN: SmartNIC-Assisted Full-Graph GNN Training with In-Network Embedding Predictioncited by 0
  6. 2026LGNNIC: Acceleration of Large-Scale GNN Training using SmartNICscited by 0