mlsanityData to 14 Oct 2026

Scalable Neural Network Training over Distributed Graphs

arXiv 2302.13053 ↗First posted 25 Feb 2023cs.LG cs.AI cs.IRcited by 0

Not on the map. Why?

Abstract (arXiv)

Graph neural networks (GNNs) fuel diverse machine learning tasks involving graph-structured data, ranging from predicting protein structures to serving personalized recommendations. Real-world graph data must often be stored distributed across many machines not just because of capacity constraints, but because of compliance with data residency or privacy laws. In such setups, network communication is costly and becomes the main bottleneck to train GNNs. Optimizations for distributed GNN training have targeted data-level improvements so far -- via caching, network-aware partitioning, and sub-sampling -- that work for data center-like setups where graph data is accessible to a single entity and data transfer costs are ignored. We present RETEXO, the first framework which eliminates the severe communication bottleneck in distributed GNN training while respecting any given data partitioning configuration. The key is a new training procedure, lazy message passing, that reorders the sequence of training GNN elements. RETEXO achieves 1-2 orders of magnitude reduction in network data costs compared to standard GNN training, while retaining accuracy. RETEXO scales gracefully with increasing decentralization and decreasing bandwidth. It is the first framework that can be used to train GNNs at all network decentralization levels -- including centralized data-center networks, wide area networks, proximity networks, and edge networks.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 41 of its 72 references that are in the map. First posted on arXiv 25 Feb 2023.

Builds on

41 of 72 references.

References · 41

  1. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  2. 2017Graph Attention Networkscited by 8,923
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  4. 2017Inductive Representation Learning on Large Graphscited by 6,320
  5. 2019A Comprehensive Survey on Graph Neural Networkscited by 2,919
  6. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,163
  7. 2018Graph Neural Networks: A Review of Methods and Applicationscited by 1,576
  8. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,150
  9. 2018Graph Convolutional Neural Networks for Web-Scale Recommender Systemscited by 1,138
  10. 2019Deep Leakage from Gradientscited by 1,021
  11. 2016Revisiting Semi-Supervised Learning with Graph Embeddingscited by 976
  12. 2019Towards Federated Learning at Scale: System Designcited by 832
29 more
  1. 2020Inverting Gradients -- How easy is it to break privacy in federated learning?cited by 712
  2. 2019Multi-scale Attributed Node Embeddingcited by 548
  3. 2019Local Model Poisoning Attacks to Byzantine-Robust Federated Learningcited by 493
  4. 2020Flower: A Friendly Federated Learning Research Frameworkcited by 455
  5. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 412
  6. 2018Analyzing Federated Learning through an Adversarial Lenscited by 351
  7. 2021See through Gradients: Image Batch Recovery via GradInversioncited by 326
  8. 2020Data Poisoning Attacks Against Federated Learning Systemscited by 275
  9. 2020Differentially Private Learning Needs Better Features (or Much More Data)cited by 235
  10. 2019AliGraph: A Comprehensive Graph Neural Network Platformcited by 153
  11. 2020DistDGL: Distributed Graph Neural Network Training for Billion-Scale Graphscited by 139
  12. 2021Subgraph Federated Learning with Missing Neighbor Generationcited by 132
  13. 2020Characteristic Functions on Graphs: Birds of a Feather, from Statistical Descriptors to Parametric Modelscited by 116
  14. 2018Adversarial Attacks on Node Embeddings via Graph Poisoningcited by 109
  15. 2020Robust Federated Learning: The Case of Affine Distribution Shiftscited by 102
  16. 2021Shift-Robust GNNs: Overcoming the Limitations of Localized Graph Training Datacited by 87
  17. 2020Coping with Label Shift via Distributionally Robust Optimisationcited by 70
  18. 2021LinkTeller: Recovering Private Edges from Graph Neural Networks via Influence Analysiscited by 65
  19. 2020GraphFL: A Federated Learning Framework for Semi-Supervised Node Classification on Graphscited by 64
  20. 2019Certifiable Robustness to Graph Perturbationscited by 62
  21. 2021DistGNN: Scalable Distributed Training for Large-Scale Graph Neural Networkscited by 59
  22. 2021Dorylus: Affordable, Scalable, and Accurate GNN Training with Distributed CPU Servers and Serverless Threadscited by 51
  23. 2021BGL: GPU-Efficient GNN Training by Optimizing Graph Data I/O and Preprocessingcited by 50
  24. 2022FederatedScope-GNN: Towards a Unified, Comprehensive and Efficient Package for Federated Graph Learningcited by 43
  25. 2020FeatGraph: A Flexible and Efficient Backend for Graph Neural Network Systemscited by 36
  26. 2020ASFGNN: Automated Separated-Federated Graph Neural Networkcited by 36
  27. 2021FedGL: Federated Graph Learning Framework with Global Self-Supervisioncited by 34
  28. 2020Accelerating Graph Sampling for Graph Machine Learning using GPUscited by 29
  29. 2022LPGNet: Link Private Graph Networks for Node Classificationcited by 12

Built on by

Not cited yet.