mlsanityData to 14 Oct 2026

MaxK-GNN: Extremely Fast GPU Kernel Design for Accelerating Graph Neural Networks Training

arXiv 2312.08656 ↗First posted 14 Dec 2023cs.LG cs.AI cs.DCcited by 25

Not on the map. Why?

Abstract (arXiv)

In the acceleration of deep neural network training, the GPU has become the mainstream platform. GPUs face substantial challenges on GNNs, such as workload imbalance and memory access irregularities, leading to underutilized hardware. Existing solutions such as PyG, DGL with cuSPARSE, and GNNAdvisor frameworks partially address these challenges but memory traffic is still significant. We argue that drastic performance improvements can only be achieved by the vertical optimization of algorithm and system innovations, rather than treating the speedup optimization as an "after-thought" (i.e., (i) given a GNN algorithm, designing an accelerator, or (ii) given hardware, mainly optimizing the GNN algorithm). In this paper, we present MaxK-GNN, an advanced high-performance GPU training system integrating algorithm and system innovation. (i) We introduce the MaxK nonlinearity and provide a theoretical analysis of MaxK nonlinearity as a universal approximator, and present the Compressed Balanced Sparse Row (CBSR) format, designed to store the data and index of the feature matrix after nonlinearity; (ii) We design a coalescing enhanced forward computation with row-wise product-based SpGEMM Kernel using CBSR for input feature matrix fetching and strategic placement of a sparse output accumulation buffer in shared memory; (iii) We develop an optimized backward computation with outer product-based and SSpMM Kernel. We conduct extensive evaluations of MaxK-GNN and report the end-to-end system run-time. Experiments show that MaxK-GNN system could approach the theoretical speedup limit according to Amdahl's law. We achieve comparable accuracy to SOTA GNNs, but at a significantly increased speed: 3.22/4.24 times speedup (vs. theoretical limits, 5.52/7.27 times) on Reddit compared to DGL and GNNAdvisor implementations.

Key facts. Cited by 25 papers in the map (data through 14 Oct 2026). Builds on 46 of its 76 references that are in the map. First posted on arXiv 14 Dec 2023.

Builds on

46 of 76 references.

References · 46

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  3. 2017Graph Attention Networkscited by 8,923
  4. 2017Inductive Representation Learning on Large Graphscited by 6,320
  5. 2018How Powerful are Graph Neural Networks?cited by 4,393
  6. 2019Fast Graph Representation Learning with PyTorch Geometriccited by 2,434
  7. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  8. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,163
  9. 2018Graph Convolutional Neural Networks for Web-Scale Recommender Systemscited by 1,138
  10. 2019GraphSAINT: Graph Sampling Based Inductive Learning Methodcited by 618
  11. 2019Cluster-GCN: An Efficient Algorithm for Training Deep and Large Graph Convolutional Networkscited by 541
  12. 2019Rigging the Lottery: Making All Tickets Winnerscited by 495
34 more
  1. 2020Graph Neural Networks in Recommender Systems: A Surveycited by 494
  2. 2017Scalable Training of Artificial Neural Networks with Adaptive Sparse Connectivity inspired by Network Sciencecited by 328
  3. 2021Graph Neural Network for Traffic Forecasting: A Surveycited by 261
  4. 2019Sparse Networks from Scratch: Faster Training without Losing Performancecited by 213
  5. 2013Maxout Networkscited by 187
  6. 2018A Systematic DNN Weight Pruning Framework using Alternating Direction Method of Multiplierscited by 120
  7. 2021A Unified Lottery Ticket Hypothesis for Graph Neural Networkscited by 118
  8. 2020Computing Graph Neural Networks: A Survey from Algorithms to Acceleratorscited by 81
  9. 2020Molecular graph generation with Graph Neural Networkscited by 80
  10. 2020HyGCN: A GCN Accelerator with Hybrid Architecturecited by 74
  11. 2019EnGN: A High-Throughput and Energy-Efficient Accelerator for Large Graph Neural Networkscited by 43
  12. 2022BNS-GCN: Efficient Full-Graph Training of Graph Convolutional Networks with Partition-Parallelism and Random Boundary Node Samplingcited by 43
  13. 2018Dissecting the NVIDIA Volta GPU Architecture via Microbenchmarkingcited by 35
  14. 2023AutoReP: Automatic ReLU Replacement for Fast Private Network Inferencecited by 29
  15. 2023LinGCN: Structural Linearized Graph Convolutional Network for Homomorphically Encrypted Inferencecited by 27
  16. 2023ReConTab: Regularized Contrastive Representation Learning for Tabular Datacited by 25
  17. 2022A Length Adaptive Algorithm-Hardware Co-design of Transformer on FPGA Through Sparse Attention and Dynamic Pipeliningcited by 22
  18. 2023Accel-GCN: High-Performance GPU Accelerator Design for Graph Convolution Networkscited by 22
  19. 2022I-GCN: A Graph Convolutional Network Accelerator with Runtime Locality Enhancement through Islandizationcited by 21
  20. 2022Optimizing Nitrogen Management with Deep Reinforcement Learning and Crop Simulationscited by 20
  21. 2022Towards Sparsification of Graph Neural Networkscited by 18
  22. 2021G-CoS: GNN-Accelerator Co-Search Towards Both Better Accuracy and Efficiencycited by 15
  23. 2023Advanced Large Language Model (LLM)-Driven Verilog Development: Enhancing Power, Performance, and Area Optimization in Code Synthesiscited by 14
  24. 2021GCoD: Graph Convolutional Network Acceleration via Dedicated Algorithm and Accelerator Co-Designcited by 13
  25. 2022RSC: Accelerating Graph Neural Networks Training via Randomized Sparse Computationscited by 12
  26. 2023RRNet: Towards ReLU-Reduced Neural Network for Two-party Computation Based Private Inferencecited by 12
  27. 2021HP-GNN: Generating High Throughput GNN Training Implementation on CPU-FPGA Heterogeneous Platformcited by 10
  28. 2023Improving Realistic Worst-Case Performance of NVCiM DNN Accelerators through Training with Right-Censored Gaussian Noisecited by 8
  29. 2023Muffin: A Framework Toward Multi-Dimension AI Fairness by Uniting Off-the-Shelf Modelscited by 8
  30. 2020Guiding Cascading Failure Search with Interpretable Graph Convolutional Networkcited by 7
  31. 2022Low-latency Mini-batch GNN Inference on CPU-FPGA Heterogeneous Platformcited by 7
  32. 2023PASNet: Polynomial Architecture Search Framework for Two-party Computation-based Secure Neural Network Deploymentcited by 7
  33. 2022GROW: A Row-Stationary Sparse-Dense GEMM Accelerator for Memory-Efficient Graph Convolutional Neural Networkscited by 5
  34. 2021SMASH: Sparse Matrix Atomic Scratchpad Hashingcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 21; 2025: 2; 2026: 2.

25 papers.

Sort:

By area:

  1. 2024SwitchTab: Switched Autoencoders Are Effective Tabular Learnerscited by 38
  2. 2024Time Series Modeling for Heart Rate Prediction: From ARIMA to Transformerscited by 35
  3. 2024Regional Style and Color Transfercited by 30
  4. 2024Enhance Image-to-Image Generation with LLaVA-generated Promptscited by 20
  5. 2024Optimizing Automated Picking Systems in Warehouse Robots Using Machine Learningcited by 20
  6. 2024Identification of Prognostic Biomarkers for Stage III Non-Small Cell Lung Carcinoma in Female Nonsmokers Using Machine Learningcited by 17
  7. 2025LoR-VP: Low-Rank Visual Prompting for Efficient Vision Model Adaptationcited by 17
  8. 2024Enhancing Visual Question Answering through Ranking-Based Hybrid Training and Multimodal Fusioncited by 16
  9. 2024Research on reinforcement learning based warehouse robot navigation algorithm in complex warehouse layoutcited by 14
  10. 2024Autonomous Navigation of Unmanned Vehicle Through Deep Reinforcement Learningcited by 12
  11. 2024Cross-border Commodity Pricing Strategy Optimization via Mixed Neural Network for Time Series Analysiscited by 12
  12. 2024Deep Analysis of Time Series Data for Smart Grid Startup Strategies: A Transformer-LSTM-PSO Model Approachcited by 11
13 more
  1. 2024Advanced AI Framework for Enhanced Detection and Assessment of Abdominal Trauma: Integrating 3D Segmentation with 2D CNN and RNN Modelscited by 9
  2. 2024Advanced Payment Security System:XGBoost, LightGBM and SMOTE Integratedcited by 8
  3. 2024Recording Brain Activity While Listening to Music Using Wearable EEG Devices Combined with Bidirectional Long Short-Term Memory Networkscited by 8
  4. 2024GTA-Net: An IoT-Integrated 3D Human Pose Estimation System for Real-Time Adolescent Sports Posture Correctioncited by 5
  5. 2025Intelligent logistics management robot path planning algorithm integrating transformer and GCN networkcited by 4
  6. 2024FlashSparse: Minimizing Computation Redundancy for Fast Sparse Matrix Multiplications on Tensor Corescited by 3
  7. 2024Data-Algorithm-Architecture Co-Optimization for Fair Neural Networks on Skin Lesion Datasetcited by 2
  8. 2024Key Information Retrieval to Classify the Unstructured Data Content of Preferential Trade Agreementscited by 1
  9. 2024A Self-guided Multimodal Approach to Enhancing Graph Representation Learning for Alzheimer's Diseasescited by 1
  10. 2024International Trade Flow Prediction with Bilateral Trade Provisionscited by 0
  11. 2024Cardiovascular Disease Detection By Leveraging Semi-Supervised Learningcited by 0
  12. 2026On Efficient Scaling of GNNs via IO-Aware Layers Implementationscited by 0
  13. 2026SparseDitto: An Agentic Sparse Compilation Framework through Architecture-Aware Synthesis on GPUscited by 0