mlsanityData to 7 Oct 2026

PyTorch-Direct: Enabling GPU Centric Data Access for Very Large Graph Neural Network Training with Irregular Accesses

arXiv 2101.07956 ↗First posted 20 Jan 2021cs.LG cs.PFcited by 12

Not on the map. Why?

Abstract (arXiv)

With the increasing adoption of graph neural networks (GNNs) in the machine learning community, GPUs have become an essential tool to accelerate GNN training. However, training GNNs on very large graphs that do not fit in GPU memory is still a challenging task. Unlike conventional neural networks, mini-batching input samples in GNNs requires complicated tasks such as traversing neighboring nodes and gathering their feature values. While this process accounts for a significant portion of the training time, we find existing GNN implementations using popular deep neural network (DNN) libraries such as PyTorch are limited to a CPU-centric approach for the entire data preparation step. This "all-in-CPU" approach has negative impact on the overall GNN training performance as it over-utilizes CPU resources and hinders GPU acceleration of GNN training. To overcome such limitations, we introduce PyTorch-Direct, which enables a GPU-centric data accessing paradigm for GNN training. In PyTorch-Direct, GPUs are capable of efficiently accessing complicated data structures in host memory directly without CPU intervention. Our microbenchmark and end-to-end GNN training results show that PyTorch-Direct reduces data transfer time by 47.1% on average and speeds up GNN training by up to 1.6x. Furthermore, by reducing CPU utilization, PyTorch-Direct also saves system power by 12.4% to 17.5% during training. To minimize programmer effort, we introduce a new "unified tensor" type along with necessary changes to the PyTorch memory allocator, dispatch logic, and placement rules. As a result, users need to change at most two lines of their PyTorch GNN training code for each tensor object to take advantage of PyTorch-Direct.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 14 of its 28 references that are in the map. First posted on arXiv 20 Jan 2021.

Builds on

14 of 28 references.

References · 14

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  4. 2017Graph Attention Networkscited by 8,854
  5. 2017Inductive Representation Learning on Large Graphscited by 6,284
  6. 2019A Comprehensive Survey on Graph Neural Networkscited by 2,906
  7. 2019Fast Graph Representation Learning with PyTorch Geometriccited by 2,414
  8. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,148
  9. 2018Graph Neural Networks: A Review of Methods and Applicationscited by 1,571
  10. 2018Graph Convolutional Neural Networks for Web-Scale Recommender Systemscited by 1,133
  11. 2019GraphSAINT: Graph Sampling Based Inductive Learning Methodcited by 618
  12. 2019Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networkscited by 540
2 more
  1. 2020Graph Neural Networks in TensorFlow and Keras with Spektralcited by 59
  2. 2020EMOGI: Efficient Memory-access for Out-of-memory Graph-traversal In GPUscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 3; 2023: 5; 2024: 1; 2025: 1; 2026: 1.

12 papers.

  1. 2023IGB: Addressing The Gaps In Labeling, Features, Heterogeneity, and Size of Public Graph Datasets for Deep Learning Researchcited by 29
  2. 2022MariusGNN: Resource-Efficient Out-of-Core Training of Graph Neural Networkscited by 19
  3. 2021TC-GNN: Bridging Sparse GNN Computation and Dense Tensor Cores on GPUscited by 17
  4. 2022MGG: Accelerating Graph Neural Networks with Fine-grained intra-kernel Communication-Computation Pipelining on Multi-GPU Platformscited by 16
  5. 2023Quiver: Supporting GPUs for Low-Latency, High-Throughput GNN Serving with Workload Awarenesscited by 11
  6. 2023Accelerating Sampling and Aggregation Operations in GNN Frameworks with GPU Initiated Direct Storage Accessescited by 6
  7. 2023FreshGNN: Reducing Memory Access via Stable Historical Embeddings for Graph Neural Network Trainingcited by 4
  8. 2022DGI: Easy and Efficient Inference for GNNscited by 2
  9. 2023Russo-Ukrainian War: Prediction and explanation of Twitter suspensioncited by 1
  10. 2024LSM-GNN: Large-scale Storage-based Multi-GPU GNN Training by Optimizing Data Transfer Schemecited by 1
  11. 2025Armada: Memory-Efficient Distributed Training of Large-Scale Graph Neural Networkscited by 0
  12. 2026MQ-GNN: A Multi-Queue Pipelined Architecture for Scalable and Efficient GNN Trainingcited by 0