# Dual-side Sparse Tensor Core

arXiv: https://arxiv.org/abs/2105.09564 · HTML: https://mlsanity.com/paper/2105.09564/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 75 references that are in the map.
- First posted on arXiv 20 May 2021.

## Builds on (most cited first, 30 of 40)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Mask R-CNN](https://mlsanity.com/paper/1703.06870/) (2017)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Effective Approaches to Attention-based Neural Machine Translation](https://mlsanity.com/paper/1508.04025/) (2015)
- [To prune, or not to prune: exploring the efficacy of pruning for model compression](https://mlsanity.com/paper/1710.01878/) (2017)
- [Learning Structured Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1608.03665/) (2016)
- [Movement Pruning: Adaptive Sparsity by Fine-Tuning](https://mlsanity.com/paper/2005.07683/) (2020)
- [LSTM: A Search Space Odyssey](https://mlsanity.com/paper/1503.04069/) (2015)
- [EIE: Efficient Inference Engine on Compressed Deep Neural Network](https://mlsanity.com/paper/1602.01528/) (2016)
- [cuDNN: Efficient Primitives for Deep Learning](https://mlsanity.com/paper/1410.0759/) (2014)
- [Accelerating Sparse Deep Neural Networks](https://mlsanity.com/paper/2104.08378/) (2021)
- [Efficient Neural Audio Synthesis](https://mlsanity.com/paper/1802.08435/) (2018)
- [SCNN: An Accelerator for Compressed-sparse Convolutional Neural Networks](https://mlsanity.com/paper/1708.04485/) (2017)
- [Spatially Adaptive Computation Time for Residual Networks](https://mlsanity.com/paper/1612.02297/) (2016)
- [Fast Algorithms for Convolutional Neural Networks](https://mlsanity.com/paper/1509.09308/) (2015)
- [Quasi-Recurrent Neural Networks](https://mlsanity.com/paper/1611.01576/) (2016)
- [More is Less: A More Complicated Network with Less Inference Complexity](https://mlsanity.com/paper/1703.08651/) (2017)
- [SBNet: Sparse Blocks Network for Fast Inference](https://mlsanity.com/paper/1801.02108/) (2018)
- [ESE: Efficient Speech Recognition Engine with Sparse LSTM on FPGA](https://mlsanity.com/paper/1612.00694/) (2016)
- [SpArch: Efficient Architecture for Sparse Matrix Multiplication](https://mlsanity.com/paper/2002.08947/) (2020)
- [Deep Learning using Rectified Linear Units (ReLU)](https://mlsanity.com/paper/1803.08375/) (2018)
- [NullHop: A Flexible Convolutional Neural Network Accelerator Based on Sparse Representations of Feature Maps](https://mlsanity.com/paper/1706.01406/) (2017)
- [Accelerating Sparse DNN Models without Hardware-Support via Tile-Wise Sparsity](https://mlsanity.com/paper/2008.13006/) (2020)
- [Deep Roots: Improving CNN Efficiency with Hierarchical Filter Groups](https://mlsanity.com/paper/1605.06489/) (2016)
- [RON: Reverse Connection with Objectness Prior Networks for Object Detection](https://mlsanity.com/paper/1707.01691/) (2017)
- [Balanced Sparsity for Efficient DNN Inference on GPU](https://mlsanity.com/paper/1811.00206/) (2018)
- [Adversarial Defense Through Network Profiling Based Path Extraction](https://mlsanity.com/paper/1904.08089/) (2019)

## Built on by (most cited first, 29 of 29)

- [OliVe: Accelerating Large Language Models via Hardware-friendly Outlier-Victim Pair Quantization](https://mlsanity.com/paper/2304.07493/) (2023)
- [SQuant: On-the-Fly Data-Free Quantization via Diagonal Hessian Approximation](https://mlsanity.com/paper/2202.07471/) (2022)
- [ANT: Exploiting Adaptive Numerical Data Type for Low-bit Deep Neural Network Quantization](https://mlsanity.com/paper/2208.14286/) (2022)
- [TorchSparse++: Efficient Training and Inference Framework for Sparse Convolution on GPUs](https://mlsanity.com/paper/2311.12862/) (2023)
- [A Survey: Collaborative Hardware and Software Design in the Era of Large Language Models](https://mlsanity.com/paper/2410.07265/) (2024)
- [BitMoD: Bit-serial Mixture-of-Datatype LLM Acceleration](https://mlsanity.com/paper/2411.11745/) (2024)
- [Ditto: Accelerating Diffusion Model via Temporal Value Similarity](https://mlsanity.com/paper/2501.11211/) (2025)
- [M-ANT: Efficient Low-bit Group Quantization for LLMs via Mathematically Adaptive Numerical Type](https://mlsanity.com/paper/2502.18755/) (2025)
- [Two Sparsities Are Better Than One: Unlocking the Performance Benefits of Sparse-Sparse Networks](https://mlsanity.com/paper/2112.13896/) (2021)
- [Sparseloop: An Analytical Approach To Sparse Tensor Accelerator Modeling](https://mlsanity.com/paper/2205.05826/) (2022)
- [vTensor: Flexible Virtual Tensor Management for Efficient LLM Serving](https://mlsanity.com/paper/2407.15309/) (2024)
- [Reconsidering the Energy Efficiency of Spiking Neural Networks Inference from Analytical Perspectives](https://mlsanity.com/paper/2409.08290/) (2024)
- [HighLight: Efficient and Flexible DNN Acceleration with Hierarchical Structured Sparsity](https://mlsanity.com/paper/2305.12718/) (2023)
- [Enabling Unstructured Sparse Acceleration on Structured Sparse Accelerators](https://mlsanity.com/paper/2403.07953/) (2024)
- [SGCN: Exploiting Compressed-Sparse Features in Deep Graph Convolutional Network Accelerators](https://mlsanity.com/paper/2301.10388/) (2023)
- [SPADE: Sparse Pillar-based 3D Object Detection Accelerator for Autonomous Driving](https://mlsanity.com/paper/2305.07522/) (2023)
- [Sparse Refinement for Efficient High-Resolution Semantic Segmentation](https://mlsanity.com/paper/2407.19014/) (2024)
- [BBS: Bi-directional Bit-level Sparsity for Deep Learning Acceleration](https://mlsanity.com/paper/2409.05227/) (2024)
- [Nesting Forward Automatic Differentiation for Memory-Efficient Deep Neural Network Training](https://mlsanity.com/paper/2209.10778/) (2022)
- [CRISP: Hybrid Structured Sparsity for Class-aware Model Pruning](https://mlsanity.com/paper/2311.14272/) (2023)
- [Slice-and-Forge: Making Better Use of Caches for Graph Convolutional Network Accelerators](https://mlsanity.com/paper/2301.09813/) (2023)
- [AdaptGear: Accelerating GNN Training via Adaptive Subgraph-Level Kernels on GPUs](https://mlsanity.com/paper/2305.17408/) (2023)
- [VeriLocc: End-to-End Cross-Architecture Register Allocation via LLM](https://mlsanity.com/paper/2506.17506/) (2025)
- [SPA-GCN: Efficient and Flexible GCN Accelerator with an Application for Graph Similarity Computation](https://mlsanity.com/paper/2111.05936/) (2021)
- [Rosko: Row Skipping Outer Products for Sparse Matrix Multiplication Kernels](https://mlsanity.com/paper/2307.03930/) (2023)
- [FLAASH: Flexible Accelerator Architecture for Sparse High-Order Tensor Contraction](https://mlsanity.com/paper/2404.16317/) (2024)
- [Dual sparse training framework: inducing activation map sparsity via Transformed ℓ1 regularization](https://mlsanity.com/paper/2405.19652/) (2024)
- [SparseMap: A Sparse Tensor Accelerator Framework Based on Evolution Strategy](https://mlsanity.com/paper/2508.12906/) (2025)
- [Celty: SpMSpV GPU Kernel and SIMT Co-Design for Efficient Dual-Sparse LLM Inference](https://mlsanity.com/paper/2608.01536/) (2026)

## Abstract (arXiv)

Leveraging sparsity in deep neural network (DNN) models is promising for accelerating model inference. Yet existing GPUs can only leverage the sparsity from weights but not activations, which are dynamic, unpredictable, and hence challenging to exploit. In this work, we propose a novel architecture to efficiently harness the dual-side sparsity (i.e., weight and activation sparsity). We take a systematic approach to understand the (dis)advantages of previous sparsity-related architectures and propose a novel, unexplored paradigm that combines outer-product computation primitive and bitmap-based encoding format. We demonstrate the feasibility of our design with minimal changes to the existing production-scale inner-product-based Tensor Core. We propose a set of novel ISA extensions and co-design the matrix-matrix multiplication and convolution algorithms, which are the two dominant computation patterns in today's DNN models, to exploit our new dual-side sparse Tensor Core. Our evaluation shows that our design can fully unleash the dual-side DNN sparsity and improve the performance by up to one order of magnitude with hlsmall hardware overhead.