mlsanityData to 14 Oct 2026

Cached Operator Reordering: A Unified View for Fast GNN Training

arXiv 2308.12093 ↗First posted 23 Aug 2023cs.LG cs.PFcited by 5

Not on the map. Why?

Abstract (arXiv)

Graph Neural Networks (GNNs) are a powerful tool for handling structured graph data and addressing tasks such as node classification, graph classification, and clustering. However, the sparse nature of GNN computation poses new challenges for performance optimization compared to traditional deep neural networks. We address these challenges by providing a unified view of GNN computation, I/O, and memory. By analyzing the computational graphs of the Graph Convolutional Network (GCN) and Graph Attention (GAT) layers -- two widely used GNN layers -- we propose alternative computation strategies. We present adaptive operator reordering with caching, which achieves a speedup of up to 2.43x for GCN compared to the current state-of-the-art. Furthermore, an exploration of different caching schemes for GAT yields a speedup of up to 1.94x. The proposed optimizations save memory, are easily implemented across various hardware platforms, and have the potential to alleviate performance bottlenecks in training large-scale GNN models.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 35 of its 48 references that are in the map. First posted on arXiv 23 Aug 2023.

Builds on

35 of 48 references.

References · 35

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  3. 2017Graph Attention Networkscited by 8,923
  4. 2017Inductive Representation Learning on Large Graphscited by 6,320
  5. 2017PointNet++: Deep Hierarchical Feature Learning on Point Sets in a Metric Spacecited by 4,401
  6. 2018How Powerful are Graph Neural Networks?cited by 4,393
  7. 2019A Comprehensive Survey on Graph Neural Networkscited by 2,919
  8. 2019Fast Graph Representation Learning with PyTorch Geometriccited by 2,434
  9. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,163
  10. 2018Graph Neural Networks: A Review of Methods and Applicationscited by 1,576
  11. 2019SuperGlue: Learning Feature Matching with Graph Neural Networkscited by 1,443
  12. 2016Geometric deep learning: going beyond Euclidean datacited by 1,048
23 more
  1. 2016Revisiting Semi-Supervised Learning with Graph Embeddingscited by 976
  2. 2019Graph Neural Networks for Social Recommendationcited by 714
  3. 2019GraphSAINT: Graph Sampling Based Inductive Learning Methodcited by 618
  4. 2017Representation Learning on Graphs: Methods and Applicationscited by 457
  5. 2018Deep Learning on Graphs: A Surveycited by 393
  6. 2020Point-GNN: Graph Neural Network for 3D Object Detection in a Point Cloudcited by 335
  7. 2020Machine Learning on Graphs: A Model and Comprehensive Taxonomycited by 156
  8. 2020Sparse GPU Kernels for Deep Learningcited by 141
  9. 2022SparseTIR: Composable Abstractions for Sparse Compilation in Deep Learningcited by 47
  10. 2020GE-SpMM: General-purpose Sparse Matrix-Matrix Multiplication on GPUs for Graph Neural Networkscited by 42
  11. 2022Parallel and Distributed Graph Neural Networks: An In-Depth Concurrency Analysiscited by 29
  12. 2021Understanding GNN Computational Graph: A Coordinated Computation, IO, and Memory Perspectivecited by 28
  13. 2019Stateful Dataflow Multigraphs: A Data-Centric Model for Performance Portability on Heterogeneous Architecturescited by 21
  14. 2021Motif Prediction with Graph Neural Networkscited by 18
  15. 2016Scaling betweenness centrality using communication-efficient sparse matrix multiplicationcited by 11
  16. 2020To Push or To Pull: On Reducing Communication and Synchronization in Graph Computationscited by 11
  17. 2019Communication-Efficient Jaccard Similarity for High-Performance Distributed Genome Comparisonscited by 10
  18. 2021Learning Combinatorial Node Labeling Algorithmscited by 10
  19. 2021Distributed Representations of Atoms and Materials for Machine Learningcited by 8
  20. 2020SlimSell: A Vectorizable Graph Representation for Breadth-First Searchcited by 7
  21. 2013A unified sparse matrix data format for efficient general sparse matrix-vector multiply on modern processors with wide SIMD unitscited by 6
  22. 2020Efficient Sparse-Dense Matrix-Matrix Multiplication on GPUs Using the Customized Sparse Storage Formatcited by 5
  23. 2021A Data-Centric Optimization Framework for Machine Learningcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 1.

5 papers.

  1. 2024Demystifying Higher-Order Graph Neural Networkscited by 4
  2. 2025Libra: Unleashing GPU Heterogeneity for High-Performance Sparse Matrix Multiplicationcited by 1
  3. 2025Higher-Order Graph Databasescited by 0
  4. 2025HePGA: A Heterogeneous Processing-in-Memory based GNN Training Acceleratorcited by 0
  5. 2026On Efficient Scaling of GNNs via IO-Aware Layers Implementationscited by 0