mlsanityData to 7 Oct 2026

Distributed Training of Large Graph Neural Networks with Variable Communication Rates

arXiv 2406.17611 ↗First posted 25 Jun 2024cs.LG eess.SPcited by 0

Not on the map. Why?

Abstract (arXiv)

Training Graph Neural Networks (GNNs) on large graphs presents unique challenges due to the large memory and computing requirements. Distributed GNN training, where the graph is partitioned across multiple machines, is a common approach to training GNNs on large graphs. However, as the graph cannot generally be decomposed into small non-interacting components, data communication between the training machines quickly limits training speeds. Compressing the communicated node activations by a fixed amount improves the training speeds, but lowers the accuracy of the trained GNN. In this paper, we introduce a variable compression scheme for reducing the communication volume in distributed GNN training without compromising the accuracy of the learned model. Based on our theoretical analysis, we derive a variable compression method that converges to a solution equivalent to the full communication case, for all graph partitioning schemes. Our empirical results show that our method attains a comparable performance to the one obtained with full communication. We outperform full communication at any fixed compression ratio for any communication budget.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 42 of its 76 references that are in the map. First posted on arXiv 25 Jun 2024.

Builds on

42 of 76 references.

References · 42

  1. 2017Inductive Representation Learning on Large Graphscited by 6,284
  2. 2018How Powerful are Graph Neural Networks?cited by 4,355
  3. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  4. 2019A Comprehensive Survey on Graph Neural Networkscited by 2,906
  5. 2018Graph Neural Networks: A Review of Methods and Applicationscited by 1,571
  6. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  7. 2016Geometric deep learning: going beyond Euclidean datacited by 1,042
  8. 2021E(n) Equivariant Graph Neural Networkscited by 951
  9. 2019Towards Federated Learning at Scale: System Designcited by 826
  10. 2019FedPAQ: A Communication-Efficient Federated Learning Method with Periodic Averaging and Quantizationcited by 349
  11. 2020Improving Graph Neural Network Expressivity via Subgraph Isomorphism Countingcited by 312
  12. 2022GraphCast: Learning skillful medium-range global weather forecastingcited by 261
30 more
  1. 2019Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communicationcited by 218
  2. 2019Universal Invariant and Equivariant Graph Neural Networkscited by 182
  3. 2019On the equivalence between graph isomorphism testing and function approximation with GNNscited by 172
  4. 2020DistDGL: Distributed Graph Neural Network Training for Billion-Scale Graphscited by 139
  5. 2019Stability Properties of Graph Neural Networkscited by 125
  6. 2020Graphon Neural Networks and the Transferability of Graph Neural Networkscited by 117
  7. 2018Convolutional Neural Network Architectures for Signals Supported on Graphscited by 94
  8. 2022Decoupling the Depth and Scope of Graph Neural Networkscited by 94
  9. 2020Degree-Quant: Quantization-Aware Training for Graph Neural Networkscited by 93
  10. 2018Error Compensated Quantized SGD and its Applications to Large-scale Distributed Optimizationcited by 76
  11. 2021DistGNN: Scalable Distributed Training for Large-Scale Graph Neural Networkscited by 59
  12. 2022Generalization Analysis of Message Passing Neural Networks on Large Random Graphscited by 58
  13. 2021Transferability of Graph Neural Networks: an Extended Graphon Approachcited by 51
  14. 2020A Linearly Convergent Algorithm for Decentralized Optimization: Sending Less Bits for Free!cited by 41
  15. 2021VQ-GNN: A Universal Framework to Scale up Graph Neural Networks using Vector Quantizationcited by 39
  16. 2020Binary Graph Neural Networkscited by 32
  17. 2020SGQuant: Squeezing the Last Bit on Graph Neural Networks with Specialized Quantizationcited by 25
  18. 2021Learning by Transference: Training Graph Neural Networks on Growing Graphscited by 22
  19. 2021Sequential Aggregation and Rematerialization: Distributed Full-batch Training of Graph Neural Networks on Large Graphscited by 18
  20. 2023Communication-Efficient Graph Neural Networks with Probabilistic Neighborhood Expansion Analysis and Cachingcited by 16
  21. 2024A Manifold Perspective on the Statistical Generalization of Graph Neural Networkscited by 16
  22. 2021Finite-Bit Quantization For Distributed Algorithms With Linear Convergencecited by 15
  23. 2022Graph Neural Networks for Multi-Robot Active Information Acquisitioncited by 12
  24. 2023Adaptive Message Quantization and Parallelization for Distributed Full-graph GNN Trainingcited by 10
  25. 2022Quantization for decentralized learning under subspace constraintscited by 7
  26. 2022Training Graph Neural Networks on Growing Stochastic Graphscited by 5
  27. 2021Distributed Adaptive Learning Under Communication Constraintscited by 4
  28. 2022BiFeat: Supercharge GNN Training via Graph Feature Quantizationcited by 4
  29. 2023Compressed Regression over Adaptive Networkscited by 2
  30. 2023FastSample: Accelerating Distributed Graph Neural Network Training for Billion-Scale Graphscited by 1

Built on by

Not cited yet.