mlsanityData to 7 Oct 2026

VENOM: A Vectorized N:M Format for Unleashing the Power of Sparse Tensor Cores

arXiv 2310.02065 ↗First posted 3 Oct 2023cs.DC cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

The increasing success and scaling of Deep Learning models demands higher computational efficiency and power. Sparsification can lead to both smaller models as well as higher compute efficiency, and accelerated hardware is becoming available. However, exploiting it efficiently requires kernel implementations, pruning algorithms, and storage formats, to utilize hardware support of specialized sparse vector units. An example of those are the NVIDIA's Sparse Tensor Cores (SPTCs), which promise a 2x speedup. However, SPTCs only support the 2:4 format, limiting achievable sparsity ratios to 50%. We present the V:N:M format, which enables the execution of arbitrary N:M ratios on SPTCs. To efficiently exploit the resulting format, we propose Spatha, a high-performance sparse-library for DL routines. We show that Spatha achieves up to 37x speedup over cuBLAS. We also demonstrate a second-order pruning technique that enables sparsification to high sparsity ratios with V:N:M and little to no loss in accuracy in modern transformers.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 26 of its 42 references that are in the map. First posted on arXiv 3 Oct 2023.

Builds on

26 of 42 references.

References · 26

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  6. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  7. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  8. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
  9. 2019Are Sixteen Heads Really Better than One?cited by 803
  10. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  11. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
  12. 2019The State of Sparsity in Deep Neural Networkscited by 471
14 more
  1. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  2. 2021Accelerating Sparse Deep Neural Networkscited by 213
  3. 2019Structured Pruning of Large Language Modelscited by 201
  4. 2021Block Pruning For Faster Transformerscited by 176
  5. 2020Sparse GPU Kernels for Deep Learningcited by 138
  6. 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
  7. 2021BERT Busters: Outlier Dimensions that Disrupt Transformerscited by 110
  8. 2020Data Movement Is All You Need: A Case Study on Optimizing Transformerscited by 108
  9. 2022PLATON: Pruning Large Transformer Models with Upper Confidence Bound of Weight Importancecited by 79
  10. 2021M-FAC: Efficient Matrix-Free Approximations of Second-Order Informationcited by 54
  11. 2022Efficient Quantized Sparse Matrix Operations on Tensor Corescited by 16
  12. 2022Dynamic N:M Fine-grained Structured Sparse Attention Mechanismcited by 14
  13. 2022GMP*: Well-Tuned Gradual Magnitude Pruning Can Outperform Most BERT-Pruning Methodscited by 14
  14. 2022Dissecting Tensor Cores via Microbenchmarks: Latency, Throughput and Numeric Behaviorscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 6; 2026: 3.

13 papers.

Sort:

By area:

  1. 2024RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptationcited by 48
  2. 2025MACKO: Sparse Matrix-Vector Multiplication for Low Sparsitycited by 6
  3. 2024Navigating Extremes: Dynamic Sparsity in Large Output Spacescited by 3
  4. 2025Lightweight Software Kernels and Hardware Extensions for Efficient Sparse Deep Neural Networks on Microcontrollerscited by 2
  5. 2025Samoyeds: Accelerating MoE Models with Structured Sparsity Leveraging Sparse Tensor Corescited by 2
  6. 2025BLaST: High Performance Inference and Pretraining using BLock Sparse Transformerscited by 2
  7. 2024Beyond 2:4: exploring V:N:M sparsity for efficient transformer inference on GPUscited by 1
  8. 2024Toward Efficient Permutation for Hierarchical N:M Sparsity on GPUscited by 0
  9. 2025TSENOR: Highly-Efficient Algorithm for Finding Transposable N:M Sparse Maskscited by 0
  10. 2025Effective Model Pruning: Measure The Redundancy of Model Componentscited by 0
  11. 2026To 2:4 Sparsity and Beyond: Neuron-level Activation Function to Accelerate LLM Pre-Trainingcited by 0
  12. 2026HieraSparse: Hierarchical Semi-Structured Sparse KV Attentioncited by 0
1 more
  1. 2026HASTE: Hardware-Aware Dynamic Sparse Training for Large Output Spacescited by 0