# High Performance Unstructured SpMM Computation Using Tensor Cores

arXiv: https://arxiv.org/abs/2408.11551 · HTML: https://mlsanity.com/paper/2408.11551/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Aug 2024.

## Built on by (most cited first, 8 of 8)

- [Dynamic Sparse Training of Diagonally Sparse Networks](https://mlsanity.com/paper/2506.11449/) (2025)
- [BLaST: High Performance Inference and Pretraining using BLock Sparse Transformers](https://mlsanity.com/paper/2507.03117/) (2025)
- [Fused3S: Fast Sparse Attention on Tensor Cores](https://mlsanity.com/paper/2505.08098/) (2025)
- [Libra: Unleashing GPU Heterogeneity for High-Performance Sparse Matrix Multiplication](https://mlsanity.com/paper/2506.22714/) (2025)
- [EntryPrune: Neural Network Feature Selection using First Impressions](https://mlsanity.com/paper/2410.02344/) (2024)
- [Accelerating GPU Inference of Large Language Models with Moderately Unstructured Sparse Weight Matrices](https://mlsanity.com/paper/2607.08786/) (2026)
- [Unified Static-Dynamic Pruning for Efficient LLM Inference](https://mlsanity.com/paper/2607.21985/) (2026)
- [UnionSparse: An Index-Efficient Sparsity Framework for Low-Bit Sparse LLM Inference on Edge](https://mlsanity.com/paper/2608.09291/) (2026)

## Abstract (arXiv)

High-performance sparse matrix-matrix (SpMM) multiplication is paramount for science and industry, as the ever-increasing sizes of data prohibit using dense data structures. Yet, existing hardware, such as Tensor Cores (TC), is ill-suited for SpMM, as it imposes strict constraints on data structures that cannot be met by unstructured sparsity found in many applications. To address this, we introduce (S)parse (Ma)trix Matrix (T)ensor Core-accelerated (SMaT): a novel SpMM library that utilizes TCs for unstructured sparse matrices. Our block-sparse library leverages the low-level CUDA MMA (matrix-matrix-accumulate) API, maximizing the performance offered by modern GPUs. Algorithmic optimizations such as sparse matrix permutation further improve performance by minimizing the number of non-zero blocks. The evaluation on NVIDIA A100 shows that SMaT outperforms SotA libraries (DASP, cuSPARSE, and Magicube) by up to 125x (on average 2.6x). SMaT can be used to accelerate many workloads in scientific computing, large-model training, inference, and others.