mlsanityData to 7 Oct 2026

Single-GPU GNN Systems: Traps and Pitfalls

arXiv 2402.03548 ↗First posted 5 Feb 2024cs.LG cs.DCcited by 1

Not on the map. Why?

Abstract (arXiv)

The current graph neural network (GNN) systems have established a clear trend of not showing training accuracy results, and directly or indirectly relying on smaller datasets for evaluations majorly. Our in-depth analysis shows that it leads to a chain of pitfalls in the system design and evaluation process, questioning the practicality of many of the proposed system optimizations, and affecting conclusions and lessons learned. We analyze many single-GPU systems and show the fundamental impact of these pitfalls. We further develop hypotheses, recommendations, and evaluation methodologies, and provide future directions. Finally, a new reference system is developed to establish a new line of optimizations rooted in solving the system-design pitfalls efficiently and practically. The proposed design can productively be integrated into prior works, thereby truly advancing the state-of-the-art.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 34 of its 55 references that are in the map. First posted on arXiv 5 Feb 2024.

Builds on

34 of 55 references.

References · 34

  1. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  2. 2017Graph Attention Networkscited by 8,854
  3. 2017Inductive Representation Learning on Large Graphscited by 6,284
  4. 2019Fast Graph Representation Learning with PyTorch Geometriccited by 2,414
  5. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  6. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  7. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  8. 2017Modeling Relational Data with Graph Convolutional Networkscited by 1,684
  9. 2019Generating Long Sequences with Sparse Transformerscited by 1,523
  10. 2018Rethinking the Value of Network Pruningcited by 595
  11. 2017Residual Gated Graph ConvNetscited by 341
  12. 2019Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networkscited by 241
22 more
  1. 2019Are We Really Making Much Progress? A Worrying Analysis of Recent Neural Recommendation Approachescited by 181
  2. 2018GaAN: Gated Attention Networks for Learning on Large and Spatiotemporal Graphscited by 164
  3. 2020DistDGL: Distributed Graph Neural Network Training for Billion-Scale Graphscited by 139
  4. 2020Dos and Don'ts of Machine Learning in Computer Securitycited by 98
  5. 2020HyGCN: A GCN Accelerator with Hybrid Architecturecited by 73
  6. 2021DistGNN: Scalable Distributed Training for Large-Scale Graph Neural Networkscited by 59
  7. 2020GNNAdvisor: An Adaptive and Efficient Runtime System for GNN Acceleration on GPUscited by 51
  8. 2021Dorylus: Affordable, Scalable, and Accurate GNN Training with Distributed CPU Servers and Serverless Threadscited by 51
  9. 2020Reducing Communication in Graph Neural Network Trainingcited by 50
  10. 2021BGL: GPU-Efficient GNN Training by Optimizing Graph Data I/O and Preprocessingcited by 50
  11. 2022SparseTIR: Composable Abstractions for Sparse Compilation in Deep Learningcited by 47
  12. 2019EnGN: A High-Throughput and Energy-Efficient Accelerator for Large Graph Neural Networkscited by 42
  13. 2020GE-SpMM: General-purpose Sparse Matrix-Matrix Multiplication on GPUs for Graph Neural Networkscited by 42
  14. 2021Accelerating Training and Inference of Graph Neural Networks with Fast Sampling and Pipeliningcited by 39
  15. 2020FeatGraph: A Flexible and Efficient Backend for Graph Neural Network Systemscited by 36
  16. 2020Accelerating Graph Sampling for Graph Machine Learning using GPUscited by 29
  17. 2021Understanding GNN Computational Graph: A Coordinated Computation, IO, and Memory Perspectivecited by 28
  18. 2020Characterizing and Understanding GCNs on GPUcited by 22
  19. 2022MariusGNN: Resource-Efficient Out-of-Core Training of Graph Neural Networkscited by 19
  20. 2021Efficient Graph Deep Learning in TensorFlow with tf_geometriccited by 17
  21. 2021TC-GNN: Bridging Sparse GNN Computation and Dense Tensor Cores on GPUscited by 17
  22. 2022MGG: Accelerating Graph Neural Networks with Fine-grained intra-kernel Communication-Computation Pipelining on Multi-GPU Platformscited by 16

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024GNNBENCH: Fair and Productive Benchmarking for Single-GPU GNN Systemcited by 3