# Adaptive Message Quantization and Parallelization for Distributed Full-graph GNN Training

arXiv: https://arxiv.org/abs/2306.01381 · HTML: https://mlsanity.com/paper/2306.01381/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 45 references that are in the map.
- First posted on arXiv 2 Jun 2023.

## Builds on (most cited first, 29 of 29)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Neural Message Passing for Quantum Chemistry](https://mlsanity.com/paper/1704.01212/) (2017)
- [A Comprehensive Survey on Graph Neural Networks](https://mlsanity.com/paper/1901.00596/) (2019)
- [Open Graph Benchmark: Datasets for Machine Learning on Graphs](https://mlsanity.com/paper/2005.00687/) (2020)
- [Link Prediction Based on Graph Neural Networks](https://mlsanity.com/paper/1802.09691/) (2018)
- [BinaryConnect: Training Deep Neural Networks with binary weights during propagations](https://mlsanity.com/paper/1511.00363/) (2015)
- [FastGCN: Fast Learning with Graph Convolutional Networks via Importance Sampling](https://mlsanity.com/paper/1801.10247/) (2018)
- [Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks](https://mlsanity.com/paper/1905.07953/) (2019)
- [Deep Learning on Graphs: A Survey](https://mlsanity.com/paper/1812.04202/) (2018)
- [Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networks](https://mlsanity.com/paper/1909.01315/) (2019)
- [Degree-Quant: Quantization-Aware Training for Graph Neural Networks](https://mlsanity.com/paper/2008.05000/) (2020)
- [Towards Unified INT8 Training for Convolutional Neural Network](https://mlsanity.com/paper/1912.12607/) (2019)
- [Error Compensated Quantized SGD and its Applications to Large-scale Distributed Optimization](https://mlsanity.com/paper/1806.08054/) (2018)
- [ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Training](https://mlsanity.com/paper/2104.14129/) (2021)
- [Minimal Variance Sampling with Provable Guarantees for Fast Training of Graph Neural Networks](https://mlsanity.com/paper/2006.13866/) (2020)
- [A Statistical Framework for Low-bitwidth Training of Deep Neural Networks](https://mlsanity.com/paper/2010.14298/) (2020)
- [Dorylus: Affordable, Scalable, and Accurate GNN Training with Distributed CPU Servers and Serverless Threads](https://mlsanity.com/paper/2105.11118/) (2021)
- [Reducing Communication in Graph Neural Network Training](https://mlsanity.com/paper/2005.03300/) (2020)
- [BGL: GPU-Efficient GNN Training by Optimizing Graph Data I/O and Preprocessing](https://mlsanity.com/paper/2112.08541/) (2021)
- [PipeGCN: Efficient Full-Graph Training of Graph Convolutional Networks with Pipelined Feature Communication](https://mlsanity.com/paper/2203.10428/) (2022)
- [BNS-GCN: Efficient Full-Graph Training of Graph Convolutional Networks with Partition-Parallelism and Random Boundary Node Sampling](https://mlsanity.com/paper/2203.10983/) (2022)
- [Accelerating Training and Inference of Graph Neural Networks with Fast Sampling and Pipelining](https://mlsanity.com/paper/2110.08450/) (2021)
- [SGQuant: Squeezing the Last Bit on Graph Neural Networks with Specialized Quantization](https://mlsanity.com/paper/2007.05100/) (2020)
- [On the Utility of Gradient Compression in Distributed Training Systems](https://mlsanity.com/paper/2103.00543/) (2021)
- [Double Quantization for Communication-Efficient Distributed Optimization](https://mlsanity.com/paper/1805.10111/) (2018)
- [Natasha: Faster Non-Convex Stochastic Optimization Via Strongly Non-Convex Parameter](https://mlsanity.com/paper/1702.00763/) (2017)

## Built on by (most cited first, 10 of 10)

- [Graph Meets LLMs: Towards Large Graph Models](https://mlsanity.com/paper/2308.14522/) (2023)
- [LLM-PQ: Serving LLM on Heterogeneous Clusters with Phase-Aware Partition and Adaptive Quantization](https://mlsanity.com/paper/2403.01136/) (2024)
- [Graph Neural Network Training Systems: A Performance Comparison of Full-Graph and Mini-Batch](https://mlsanity.com/paper/2406.00552/) (2024)
- [GNNPipe: Scaling Deep GNN Training with Pipelined Model Parallelism](https://mlsanity.com/paper/2308.10087/) (2023)
- [Plexus: Taming Billion-edge Graphs with 3D Parallel Full-graph GNN Training](https://mlsanity.com/paper/2505.04083/) (2025)
- [Synergistic Signals: Exploiting Co-Engagement and Semantic Links via Graph Neural Networks](https://mlsanity.com/paper/2312.04071/) (2023)
- [Distributed Training of Large Graph Neural Networks with Variable Communication Rates](https://mlsanity.com/paper/2406.17611/) (2024)
- [MixGCN: Scalable GCN Training by Mixture of Parallelism and Mixture of Accelerators](https://mlsanity.com/paper/2501.01951/) (2025)
- [Demystifying Distributed Training of Graph Neural Networks for Link Prediction](https://mlsanity.com/paper/2506.20818/) (2025)
- [SNI-GNN: SmartNIC-Assisted Full-Graph GNN Training with In-Network Embedding Prediction](https://mlsanity.com/paper/2608.06441/) (2026)

## Abstract (arXiv)

Distributed full-graph training of Graph Neural Networks (GNNs) over large graphs is bandwidth-demanding and time-consuming. Frequent exchanges of node features, embeddings and embedding gradients (all referred to as messages) across devices bring significant communication overhead for nodes with remote neighbors on other devices (marginal nodes) and unnecessary waiting time for nodes without remote neighbors (central nodes) in the training graph. This paper proposes an efficient GNN training system, AdaQP, to expedite distributed full-graph GNN training. We stochastically quantize messages transferred across devices to lower-precision integers for communication traffic reduction and advocate communication-computation parallelization between marginal nodes and central nodes. We provide theoretical analysis to prove fast training convergence (at the rate of O(T^{-1}) with T being the total number of training epochs) and design an adaptive quantization bit-width assignment scheme for each message based on the analysis, targeting a good trade-off between training convergence and efficiency. Extensive experiments on mainstream graph datasets show that AdaQP substantially improves distributed full-graph training's throughput (up to 3.01 X) with negligible accuracy drop (at most 0.30%) or even accuracy improvement (up to 0.19%) in most cases, showing significant advantages over the state-of-the-art works.