mlsanityData to 7 Oct 2026

Communication-Efficient Graph Neural Networks with Probabilistic Neighborhood Expansion Analysis and Caching

arXiv 2305.03152 ↗First posted 4 May 2023cs.LG cs.DC cs.PFcited by 16

Not on the map. Why?

Abstract (arXiv)

Training and inference with graph neural networks (GNNs) on massive graphs has been actively studied since the inception of GNNs, owing to the widespread use and success of GNNs in applications such as recommendation systems and financial forensics. This paper is concerned with minibatch training and inference with GNNs that employ node-wise sampling in distributed settings, where the necessary partitioning of vertex features across distributed storage causes feature communication to become a major bottleneck that hampers scalability. To significantly reduce the communication volume without compromising prediction accuracy, we propose a policy for caching data associated with frequently accessed vertices in remote partitions. The proposed policy is based on an analysis of vertex-wise inclusion probabilities (VIP) during multi-hop neighborhood sampling, which may expand the neighborhood far beyond the partition boundaries of the graph. VIP analysis not only enables the elimination of the communication bottleneck, but it also offers a means to organize in-memory data by prioritizing GPU storage for the most frequently accessed vertex features. We present SALIENT++, which extends the prior state-of-the-art SALIENT system to work with partitioned feature data and leverages the VIP-driven caching policy. SALIENT++ retains the local training efficiency and scalability of SALIENT by using a deep pipeline and drastically reducing communication volume while consuming only a fraction of the storage required by SALIENT. We provide experimental results with the Open Graph Benchmark data sets and demonstrate that training a 3-layer GraphSAGE model with SALIENT++ on 8 single-GPU machines is 7.1 faster than with SALIENT on 1 single-GPU machine, and 12.7 faster than with DistDGL on 8 single-GPU machines.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 27 of its 45 references that are in the map. First posted on arXiv 4 May 2023.

Builds on

27 of 45 references.

References · 27

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  3. 2017Graph Attention Networkscited by 8,854
  4. 2017Inductive Representation Learning on Large Graphscited by 6,284
  5. 2018How Powerful are Graph Neural Networks?cited by 4,355
  6. 2017Neural Message Passing for Quantum Chemistrycited by 3,460
  7. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,148
  8. 2016Optimization Methods for Large-Scale Machine Learningcited by 1,140
  9. 2018Graph Convolutional Neural Networks for Web-Scale Recommender Systemscited by 1,133
  10. 2015Gated Graph Sequence Neural Networkscited by 792
  11. 2019GraphSAINT: Graph Sampling Based Inductive Learning Methodcited by 618
  12. 2018FastGCN: Fast Learning with Graph Convolutional Networks via Importance Samplingcited by 583
15 more
  1. 2019Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networkscited by 540
  2. 2021OGB-LSC: A Large-Scale Challenge for Machine Learning on Graphscited by 328
  3. 2019Layer-Dependent Importance Sampling for Training Deep and Large Graph Convolutional Networkscited by 177
  4. 2019Anti-Money Laundering in Bitcoin: Experimenting with Graph Convolutional Networks for Financial Forensicscited by 158
  5. 2020DistDGL: Distributed Graph Neural Network Training for Billion-Scale Graphscited by 139
  6. 2021DistGNN: Scalable Distributed Training for Large-Scale Graph Neural Networkscited by 59
  7. 2020GNNAdvisor: An Adaptive and Efficient Runtime System for GNN Acceleration on GPUscited by 51
  8. 2022BNS-GCN: Efficient Full-Graph Training of Graph Convolutional Networks with Partition-Parallelism and Random Boundary Node Samplingcited by 43
  9. 2021Accelerating Training and Inference of Graph Neural Networks with Fast Sampling and Pipeliningcited by 39
  10. 2021Distributed Hybrid CPU and GPU training for Graph Neural Networks on Billion-Scale Graphscited by 33
  11. 2020Bandit Samplers for Training Graph Neural Networkscited by 30
  12. 2021Marius: Learning Massive Graph Embeddings on a Single Machinecited by 29
  13. 2021Large Graph Convolutional Network Training with GPU-Oriented Data Communication Architecturecited by 25
  14. 2021Global Neighbor Sampling for Mixed CPU-GPU Training on Giant Graphscited by 22
  15. 2018Stochastic Gradient Descent with Biased but Consistent Gradient Estimatorscited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 7; 2025: 3; 2026: 3.

16 papers.

Sort:

By area:

  1. 2024Graph Neural Preconditioners for Iterative Solutions of Sparse Linear Systemscited by 15
  2. 2024Graph Neural Network Training Systems: A Performance Comparison of Full-Graph and Mini-Batchcited by 10
  3. 2023Comprehensive Evaluation of GNN Training Systems: A Data Management Perspectivecited by 9
  4. 2024The Shape of Money Laundering: Subgraph Representation Learning on the Blockchain with the Elliptic2 Datasetcited by 7
  5. 2023Distributed Matrix-Based Sampling for Graph Neural Network Trainingcited by 6
  6. 2024MassiveGNN: Efficient Training via Prefetching for Massively Connected Distributed Graphscited by 3
  7. 2023GraNNDis: Efficient Unified Distributed Training Framework for Deep GNNs on Large Clusterscited by 2
  8. 2025Graph Learning at Scale: Characterizing and Optimizing Pre-Propagation GNNscited by 2
  9. 2025Deal: Distributed End-to-End GNN Inference for All Nodescited by 1
  10. 2024PerOS: Personalized Self-Adapting Operating Systems in the Cloudcited by 0
  11. 2024SDT-GNN: Streaming-based Distributed Training Framework for Graph Neural Networkscited by 0
  12. 2024Distributed Training of Large Graph Neural Networks with Variable Communication Ratescited by 0
4 more
  1. 2025Armada: Memory-Efficient Distributed Training of Large-Scale Graph Neural Networkscited by 0
  2. 2026Rudder: Steering Prefetching in Distributed GNN Training using LLM Agentscited by 0
  3. 2026Communication-free Sampling and 4D Hybrid Parallelism for Scalable Mini-batch GNN Trainingcited by 0
  4. 2026Taurus: Accelerating Out-of-Core Graph Neural Network Inference on Billion-Scale Graphscited by 0