High Performance Unstructured SpMM Computation Using Tensor Cores
Not on the map. Why?
Abstract (arXiv)
High-performance sparse matrix-matrix (SpMM) multiplication is paramount for science and industry, as the ever-increasing sizes of data prohibit using dense data structures. Yet, existing hardware, such as Tensor Cores (TC), is ill-suited for SpMM, as it imposes strict constraints on data structures that cannot be met by unstructured sparsity found in many applications. To address this, we introduce (S)parse (Ma)trix Matrix (T)ensor Core-accelerated (SMaT): a novel SpMM library that utilizes TCs for unstructured sparse matrices. Our block-sparse library leverages the low-level CUDA MMA (matrix-matrix-accumulate) API, maximizing the performance offered by modern GPUs. Algorithmic optimizations such as sparse matrix permutation further improve performance by minimizing the number of non-zero blocks. The evaluation on NVIDIA A100 shows that SMaT outperforms SotA libraries (DASP, cuSPARSE, and Magicube) by up to 125x (on average 2.6x). SMaT can be used to accelerate many workloads in scientific computing, large-model training, inference, and others.
Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). First posted on arXiv 21 Aug 2024.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2024: 1; 2025: 4; 2026: 3.
8 papers.
- 2025Dynamic Sparse Training of Diagonally Sparse Networkscited by 4
- 2025BLaST: High Performance Inference and Pretraining using BLock Sparse Transformerscited by 3
- 2025Fused3S: Fast Sparse Attention on Tensor Corescited by 2
- 2025Libra: Unleashing GPU Heterogeneity for High-Performance Sparse Matrix Multiplicationcited by 1
- 2024EntryPrune: Neural Network Feature Selection using First Impressionscited by 0
- 2026Accelerating GPU Inference of Large Language Models with Moderately Unstructured Sparse Weight Matricescited by 0
- 2026Unified Static-Dynamic Pruning for Efficient LLM Inferencecited by 0
- 2026UnionSparse: An Index-Efficient Sparsity Framework for Low-Bit Sparse LLM Inference on Edgecited by 0
- 2026UnionSparse: An Index-Efficient Sparsity Framework for Low-Bit Sparse LLM Inference on Edge10 Aug 2026
- 2026Unified Static-Dynamic Pruning for Efficient LLM Inference24 Jul 2026
- 2026Accelerating GPU Inference of Large Language Models with Moderately Unstructured Sparse Weight Matrices13 Jun 2026
- 2025BLaST: High Performance Inference and Pretraining using BLock Sparse Transformers3 Jul 2025
- 2025Libra: Unleashing GPU Heterogeneity for High-Performance Sparse Matrix Multiplication28 Jun 2025
- 2025Dynamic Sparse Training of Diagonally Sparse Networks13 Jun 2025
- 2025Fused3S: Fast Sparse Attention on Tensor Cores12 May 2025
- 2024EntryPrune: Neural Network Feature Selection using First Impressions3 Oct 2024