mlsanityData to 7 Oct 2026

Adaptive Message Quantization and Parallelization for Distributed Full-graph GNN Training

arXiv 2306.01381 ↗First posted 2 Jun 2023cs.LG cs.AI cs.DCcited by 10

Not on the map. Why?

Abstract (arXiv)

Distributed full-graph training of Graph Neural Networks (GNNs) over large graphs is bandwidth-demanding and time-consuming. Frequent exchanges of node features, embeddings and embedding gradients (all referred to as messages) across devices bring significant communication overhead for nodes with remote neighbors on other devices (marginal nodes) and unnecessary waiting time for nodes without remote neighbors (central nodes) in the training graph. This paper proposes an efficient GNN training system, AdaQP, to expedite distributed full-graph GNN training. We stochastically quantize messages transferred across devices to lower-precision integers for communication traffic reduction and advocate communication-computation parallelization between marginal nodes and central nodes. We provide theoretical analysis to prove fast training convergence (at the rate of O(T^{-1}) with T being the total number of training epochs) and design an adaptive quantization bit-width assignment scheme for each message based on the analysis, targeting a good trade-off between training convergence and efficiency. Extensive experiments on mainstream graph datasets show that AdaQP substantially improves distributed full-graph training's throughput (up to 3.01 X) with negligible accuracy drop (at most 0.30%) or even accuracy improvement (up to 0.19%) in most cases, showing significant advantages over the state-of-the-art works.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 29 of its 45 references that are in the map. First posted on arXiv 2 Jun 2023.

Builds on

29 of 45 references.

References · 29

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  3. 2017Inductive Representation Learning on Large Graphscited by 6,284
  4. 2018How Powerful are Graph Neural Networks?cited by 4,355
  5. 2017Neural Message Passing for Quantum Chemistrycited by 3,460
  6. 2019A Comprehensive Survey on Graph Neural Networkscited by 2,906
  7. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,148
  8. 2018Link Prediction Based on Graph Neural Networkscited by 829
  9. 2018FastGCN: Fast Learning with Graph Convolutional Networks via Importance Samplingcited by 583
  10. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
  11. 2019Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networkscited by 540
  12. 2018Deep Learning on Graphs: A Surveycited by 393
17 more
  1. 2019Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networkscited by 241
  2. 2020Degree-Quant: Quantization-Aware Training for Graph Neural Networkscited by 93
  3. 2019Towards Unified INT8 Training for Convolutional Neural Networkcited by 77
  4. 2018Error Compensated Quantized SGD and its Applications to Large-scale Distributed Optimizationcited by 76
  5. 2021ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Trainingcited by 61
  6. 2020Minimal Variance Sampling with Provable Guarantees for Fast Training of Graph Neural Networkscited by 54
  7. 2020A Statistical Framework for Low-bitwidth Training of Deep Neural Networkscited by 51
  8. 2021Dorylus: Affordable, Scalable, and Accurate GNN Training with Distributed CPU Servers and Serverless Threadscited by 51
  9. 2020Reducing Communication in Graph Neural Network Trainingcited by 50
  10. 2021BGL: GPU-Efficient GNN Training by Optimizing Graph Data I/O and Preprocessingcited by 50
  11. 2022PipeGCN: Efficient Full-Graph Training of Graph Convolutional Networks with Pipelined Feature Communicationcited by 45
  12. 2022BNS-GCN: Efficient Full-Graph Training of Graph Convolutional Networks with Partition-Parallelism and Random Boundary Node Samplingcited by 43
  13. 2021Accelerating Training and Inference of Graph Neural Networks with Fast Sampling and Pipeliningcited by 39
  14. 2021On the Utility of Gradient Compression in Distributed Training Systemscited by 26
  15. 2020SGQuant: Squeezing the Last Bit on Graph Neural Networks with Specialized Quantizationcited by 25
  16. 2018Double Quantization for Communication-Efficient Distributed Optimizationcited by 18
  17. 2017Natasha: Faster Non-Convex Stochastic Optimization Via Strongly Non-Convex Parametercited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 3; 2025: 3; 2026: 1.

10 papers.

  1. 2023Graph Meets LLMs: Towards Large Graph Modelscited by 19
  2. 2024LLM-PQ: Serving LLM on Heterogeneous Clusters with Phase-Aware Partition and Adaptive Quantizationcited by 14
  3. 2024Graph Neural Network Training Systems: A Performance Comparison of Full-Graph and Mini-Batchcited by 10
  4. 2023GNNPipe: Scaling Deep GNN Training with Pipelined Model Parallelismcited by 4
  5. 2025Plexus: Taming Billion-edge Graphs with 3D Parallel Full-graph GNN Trainingcited by 1
  6. 2023Synergistic Signals: Exploiting Co-Engagement and Semantic Links via Graph Neural Networkscited by 0
  7. 2024Distributed Training of Large Graph Neural Networks with Variable Communication Ratescited by 0
  8. 2025MixGCN: Scalable GCN Training by Mixture of Parallelism and Mixture of Acceleratorscited by 0
  9. 2025Demystifying Distributed Training of Graph Neural Networks for Link Predictioncited by 0
  10. 2026SNI-GNN: SmartNIC-Assisted Full-Graph GNN Training with In-Network Embedding Predictioncited by 0