mlsanityData to 14 Oct 2026

FlashSparse: Minimizing Computation Redundancy for Fast Sparse Matrix Multiplications on Tensor Cores

arXiv 2412.11007 ↗First posted 15 Dec 2024cs.DC cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Sparse Matrix-matrix Multiplication (SpMM) and Sampled Dense-dense Matrix Multiplication (SDDMM) are important sparse operators in scientific computing and deep learning. Tensor Core Units (TCUs) enhance modern accelerators with superior computing power, which is promising to boost the performance of matrix operators to a higher level. However, due to the irregularity of unstructured sparse data, it is difficult to deliver practical speedups on TCUs. To this end, we propose FlashSparse, a novel approach to bridge the gap between sparse workloads and the TCU architecture. Specifically, FlashSparse minimizes the sparse granularity for SpMM and SDDMM on TCUs through a novel swap-and-transpose matrix multiplication strategy. Benefiting from the minimum sparse granularity, the computation redundancy is remarkably reduced while the computing power of TCUs is fully utilized. Besides, FlashSparse is equipped with a memory-efficient thread mapping strategy for coalesced data access and a sparse matrix storage format to save memory footprint. Extensive experimental results on H100 and RTX 4090 GPUs show that FlashSparse sets a new state-of-the-art for sparse matrix multiplications (geometric mean 5.5x speedup over DTC-SpMM and 3.22x speedup over RoDe).

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 17 of its 50 references that are in the map. First posted on arXiv 15 Dec 2024.

Builds on

17 of 50 references.

References · 17

  1. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,361
  2. 2017Graph Attention Networkscited by 8,923
  3. 2019Fast Graph Representation Learning with PyTorch Geometriccited by 2,434
  4. 2021Graph Neural Network for Traffic Forecasting: A Surveycited by 261
  5. 2020Sparse GPU Kernels for Deep Learningcited by 141
  6. 2018Attention-based Graph Neural Network for Semi-supervised Learningcited by 101
  7. 2019Deep Reinforcement Learning meets Graph Neural Networks: exploring a routing optimization use casecited by 53
  8. 2020GNNAdvisor: An Adaptive and Efficient Runtime System for GNN Acceleration on GPUscited by 51
  9. 2020GE-SpMM: General-purpose Sparse Matrix-Matrix Multiplication on GPUs for Graph Neural Networkscited by 42
  10. 2023IGB: Addressing The Gaps In Labeling, Features, Heterogeneity, and Size of Public Graph Datasets for Deep Learning Researchcited by 30
  11. 2023MaxK-GNN: Extremely Fast GPU Kernel Design for Accelerating Graph Neural Networks Trainingcited by 25
  12. 2021TC-GNN: Bridging Sparse GNN Computation and Dense Tensor Cores on GPUscited by 17
5 more
  1. 2022Efficient Quantized Sparse Matrix Operations on Tensor Corescited by 16
  2. 2014Tag-Aware Ordinal Sparse Factor Analysis for Learning and Content Analyticscited by 15
  3. 2021Sparse Structure Learning via Graph Neural Networks for Inductive Document Classificationcited by 15
  4. 2021APNN-TC: Accelerating Arbitrary Precision Neural Networks on Ampere GPU Tensor Corescited by 12
  5. 2020Accelerating Sparse Matrix-Matrix Multiplication with GPU Tensor Corescited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Fused3S: Fast Sparse Attention on Tensor Corescited by 2
  2. 2025Libra: Unleashing GPU Heterogeneity for High-Performance Sparse Matrix Multiplicationcited by 1
  3. 2026Beyond Masked Sparsity: SNACK Enables Truly Sparse Neural Networks on GPUcited by 0