mlsanityData to 14 Oct 2026

Efficient Quantized Sparse Matrix Operations on Tensor Cores

arXiv 2209.06979 ↗First posted 14 Sep 2022cs.DC cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

The exponentially growing model size drives the continued success of deep learning, but it brings prohibitive computation and memory cost. From the algorithm perspective, model sparsification and quantization have been studied to alleviate the problem. From the architecture perspective, hardware vendors provide Tensor cores for acceleration. However, it is very challenging to gain practical speedups from sparse, low-precision matrix operations on Tensor cores, because of the strict requirements for data layout and lack of support for efficiently manipulating the low-precision integers. We propose Magicube, a high-performance sparse-matrix library for low-precision integers on Tensor cores. Magicube supports SpMM and SDDMM, two major sparse operations in deep learning with mixed precision. Experimental results on an NVIDIA A100 GPU show that Magicube achieves on average 1.44x (up to 2.37x) speedup over the vendor-optimized library for sparse kernels, and 1.43x speedup over the state-of-the-art with a comparable accuracy for end-to-end sparse Transformer inference.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 49 of its 79 references that are in the map. First posted on arXiv 14 Sep 2022.

Builds on

49 of 79 references.

References · 49

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,548
  8. 2020Longformer: The Long-Document Transformercited by 3,413
  9. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  10. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  11. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  12. 2021ViViT: A Video Vision Transformercited by 1,620
37 more
  1. 2020Rethinking Attention with Performerscited by 1,601
  2. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  3. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  4. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  5. 2017Mixed Precision Trainingcited by 1,156
  6. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,101
  7. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 849
  8. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  9. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  10. 2017Deep Learning Scaling is Predictable, Empiricallycited by 582
  11. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  12. 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 527
  13. 2021A White Paper on Neural Network Quantizationcited by 472
  14. 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 443
  15. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 403
  16. 2019Q8BERT: Quantized 8Bit BERTcited by 300
  17. 2016EIE: Efficient Inference Engine on Compressed Deep Neural Networkcited by 295
  18. 2018LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networkscited by 274
  19. 2021I-BERT: Integer-only BERT Quantizationcited by 255
  20. 2019A Constructive Prediction of the Generalization Error Across Scalescited by 210
  21. 2020Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluationcited by 188
  22. 2021Block Pruning For Faster Transformerscited by 177
  23. 2019Drawing Early-Bird Tickets: Towards More Efficient Training of Deep Networkscited by 160
  24. 2020Sparse GPU Kernels for Deep Learningcited by 141
  25. 2020APQ: Joint Search for Network Architecture, Pruning and Quantization Policycited by 99
  26. 2020Bayesian Bits: Unifying Quantization and Pruningcited by 94
  27. 2021Chimera: Efficiently Training Large-Scale Neural Networks with Bidirectional Pipelinescited by 68
  28. 2019Efficient 8-Bit Quantization of Transformer Neural Machine Language Translation Modelcited by 59
  29. 2020LadaBERT: Lightweight Adaptation of BERT through Hybrid Model Compressioncited by 45
  30. 2017Exploring the Regularity of Sparse Structure in Convolutional Neural Networkscited by 37
  31. 2018Large-Scale Distributed Second-Order Optimization Using Kronecker-Factored Approximate Curvature for Deep Convolutional Neural Networkscited by 37
  32. 2018Dissecting the NVIDIA Volta GPU Architecture via Microbenchmarkingcited by 35
  33. 2021KAISA: An Adaptive Second-Order Optimizer Framework for Deep Neural Networkscited by 20
  34. 2020Accelerating Binarized Neural Networks via Bit-Tensor-Cores in Turing GPUscited by 14
  35. 2021APNN-TC: Accelerating Arbitrary Precision Neural Networks on Ampere GPU Tensor Corescited by 12
  36. 2020Joint Pruning & Quantization for Extremely Sparse Neural Networkscited by 9
  37. 2020Fast and Scalable Sparse Triangular Solver for Multi-GPU Based HPC Architecturescited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 2; 2025: 7; 2026: 2.

16 papers.

Sort:

By area:

  1. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 258
  2. 2024RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptationcited by 48
  3. 2023On Efficient Training of Large-Scale Deep Learning Models: A Literature Reviewcited by 26
  4. 2023Chameleon: a Heterogeneous and Disaggregated Accelerator System for Retrieval-Augmented Language Modelscited by 23
  5. 2023Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsitycited by 17
  6. 2023VENOM: A Vectorized N:M Format for Unleashing the Power of Sparse Tensor Corescited by 13
  7. 2025Chipmunk: Training-Free Acceleration of Diffusion Transformers with Dynamic Column-Sparse Deltascited by 8
  8. 2024FlashSparse: Minimizing Computation Redundancy for Fast Sparse Matrix Multiplications on Tensor Corescited by 3
  9. 2025Efficient Mixed Precision Quantization in Graph Neural Networkscited by 3
  10. 2025BLaST: High Performance Inference and Pretraining using BLock Sparse Transformerscited by 3
  11. 2025Lightweight Software Kernels and Hardware Extensions for Efficient Sparse Deep Neural Networks on Microcontrollerscited by 2
  12. 2025Fused3S: Fast Sparse Attention on Tensor Corescited by 2
4 more
  1. 2023STen: Productive and Efficient Sparsity in PyTorchcited by 1
  2. 2025Libra: Unleashing GPU Heterogeneity for High-Performance Sparse Matrix Multiplicationcited by 1
  3. 2026MO-SAE:Multi-Objective Stacked Autoencoders Optimization for Edge Anomaly Detectioncited by 0
  4. 2026Accelerating GPU Inference of Large Language Models with Moderately Unstructured Sparse Weight Matricescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Efficient Arbitrary Precision Acceleration for Large Language Models on GPU Tensor Cores
  2. 2026SlideSparse: Fast and Flexible (2N-2):2N Structured Sparsity
  3. 2026Hardware-Native Joint Sparse-Quantization for Trillion-Scale Mixture-of-Experts
  4. 2025APT-LLM: Exploiting Arbitrary-Precision Tensor Core Computing for LLM Acceleration
  5. 2023Quantization Aware Factorization for Deep Neural Network Compression
  6. 2026To 2:4 Sparsity and Beyond: Neuron-level Activation Function to Accelerate LLM Pre-Training
  7. 2021SparseDNN: Fast Sparse Deep Learning Inference on CPUs
  8. 2025The Cambrian Explosion of Mixed-Precision Matrix Multiplication for Quantized Deep Learning Inference