# TC-GNN: Bridging Sparse GNN Computation and Dense Tensor Cores on GPUs

arXiv: https://arxiv.org/abs/2112.02052 · HTML: https://mlsanity.com/paper/2112.02052/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 70 references that are in the map.
- First posted on arXiv 3 Dec 2021.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Graph Attention Networks](https://mlsanity.com/paper/1710.10903/) (2017)
- [Inductive Representation Learning on Large Graphs](https://mlsanity.com/paper/1706.02216/) (2017)
- [How Powerful are Graph Neural Networks?](https://mlsanity.com/paper/1810.00826/) (2018)
- [Fast Graph Representation Learning with PyTorch Geometric](https://mlsanity.com/paper/1903.02428/) (2019)
- [node2vec: Scalable Feature Learning for Networks](https://mlsanity.com/paper/1607.00653/) (2016)
- [DeepWalk: Online Learning of Social Representations](https://mlsanity.com/paper/1403.6652/) (2014)
- [Link Prediction Based on Graph Neural Networks](https://mlsanity.com/paper/1802.09691/) (2018)
- [Hierarchical Graph Representation Learning with Differentiable Pooling](https://mlsanity.com/paper/1806.08804/) (2018)
- [In-Datacenter Performance Analysis of a Tensor Processing Unit](https://mlsanity.com/paper/1704.04760/) (2017)
- [Deep Graph Library: A Graph-Centric, Highly-Performant Package for Graph Neural Networks](https://mlsanity.com/paper/1909.01315/) (2019)
- [Attention-based Graph Neural Network for Semi-supervised Learning](https://mlsanity.com/paper/1803.03735/) (2018)
- [GE-SpMM: General-purpose Sparse Matrix-Matrix Multiplication on GPUs for Graph Neural Networks](https://mlsanity.com/paper/2007.03179/) (2020)
- [Learning Graph Representations with Embedding Propagation](https://mlsanity.com/paper/1710.03059/) (2017)
- [QGTC: Accelerating Quantized Graph Neural Networks via GPU Tensor Core](https://mlsanity.com/paper/2111.09547/) (2021)
- [A Broader Picture of Random-walk Based Graph Embedding](https://mlsanity.com/paper/2110.12344/) (2021)
- [Accelerating Binarized Neural Networks via Bit-Tensor-Cores in Turing GPUs](https://mlsanity.com/paper/2006.16578/) (2020)
- [PyTorch-Direct: Enabling GPU Centric Data Access for Very Large Graph Neural Network Training with Irregular Accesses](https://mlsanity.com/paper/2101.07956/) (2021)
- [Deep Feature Learning via Structured Graph Laplacian Embedding for Person Re-Identification](https://mlsanity.com/paper/1707.07791/) (2017)
- [Accelerating Sparse Matrix-Matrix Multiplication with GPU Tensor Cores](https://mlsanity.com/paper/2009.14600/) (2020)

## Built on by (most cited first, 17 of 17)

- [FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving](https://mlsanity.com/paper/2501.01005/) (2025)
- [DSV: Exploiting Dynamic Sparsity to Accelerate Large-Scale Video DiT Training](https://mlsanity.com/paper/2502.07590/) (2025)
- [GSplit: Scaling Graph Neural Network Training on Large Graphs via Split-Parallelism](https://mlsanity.com/paper/2303.13775/) (2023)
- [DiskGNN: Bridging I/O Efficiency and Model Accuracy for Out-of-Core GNN Training](https://mlsanity.com/paper/2405.05231/) (2024)
- [Efficient Arbitrary Precision Acceleration for Large Language Models on GPU Tensor Cores](https://mlsanity.com/paper/2409.17870/) (2024)
- [GNNBENCH: Fair and Productive Benchmarking for Single-GPU GNN System](https://mlsanity.com/paper/2404.04118/) (2024)
- [FlashSparse: Minimizing Computation Redundancy for Fast Sparse Matrix Multiplications on Tensor Cores](https://mlsanity.com/paper/2412.11007/) (2024)
- [Fused3S: Fast Sparse Attention on Tensor Cores](https://mlsanity.com/paper/2505.08098/) (2025)
- [Single-GPU GNN Systems: Traps and Pitfalls](https://mlsanity.com/paper/2402.03548/) (2024)
- [Optimization of GNN Training Through Half-precision](https://mlsanity.com/paper/2411.01109/) (2024)
- [Accelerating Sparse Graph Neural Networks with Tensor Core Optimization](https://mlsanity.com/paper/2412.12218/) (2024)
- [Libra: Unleashing GPU Heterogeneity for High-Performance Sparse Matrix Multiplication](https://mlsanity.com/paper/2506.22714/) (2025)
- [APT-LLM: Exploiting Arbitrary-Precision Tensor Core Computing for LLM Acceleration](https://mlsanity.com/paper/2508.19087/) (2025)
- [AutoSAGE: Input-Aware CUDA Scheduling for Sparse GNN Aggregation (SpMM/SDDMM) and CSR Attention](https://mlsanity.com/paper/2511.17594/) (2025)
- [On Efficient Scaling of GNNs via IO-Aware Layers Implementations](https://mlsanity.com/paper/2605.31500/) (2026)
- [Accelerating GPU Inference of Large Language Models with Moderately Unstructured Sparse Weight Matrices](https://mlsanity.com/paper/2607.08786/) (2026)
- [SparseDitto: An Agentic Sparse Compilation Framework through Architecture-Aware Synthesis on GPUs](https://mlsanity.com/paper/2608.05033/) (2026)

## Abstract (arXiv)

Recently, graph neural networks (GNNs), as the backbone of graph-based machine learning, demonstrate great success in various domains (e.g., e-commerce). However, the performance of GNNs is usually unsatisfactory due to the highly sparse and irregular graph-based operations. To this end, we propose TC-GNN, the first GNN acceleration framework based on GPU Tensor Core Units (TCUs). The core idea is to reconcile the "Sparse" GNN computation with the high-performance "Dense" TCUs. Specifically, we conduct an in-depth analysis of the sparse operations in mainstream GNN computing frameworks. We introduce a novel sparse graph translation technique to facilitate TCU processing of the sparse GNN workload. We implement an effective CUDA core and TCU collaboration design to fully utilize GPU resources. We integrate TC-GNN with the PyTorch framework for high programmability. Rigorous experiments show an average of 1.70X speedup over the state-of-the-art DGL framework across various models and datasets.