# Enabling Unstructured Sparse Acceleration on Structured Sparse Accelerators

arXiv: https://arxiv.org/abs/2403.07953 · HTML: https://mlsanity.com/paper/2403.07953/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 67 references that are in the map.
- First posted on arXiv 12 Mar 2024.

## Builds on (most cited first, 30 of 35)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shot](https://mlsanity.com/paper/2301.00774/) (2023)
- [Deep Learning Recommendation Model for Personalization and Recommendation Systems](https://mlsanity.com/paper/1906.00091/) (2019)
- [MLPerf Inference Benchmark](https://mlsanity.com/paper/1911.02549/) (2019)
- [Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch](https://mlsanity.com/paper/2102.04010/) (2021)
- [Primer: Searching for Efficient Transformers for Language Modeling](https://mlsanity.com/paper/2109.08668/) (2021)
- [SCNN: An Accelerator for Compressed-sparse Convolutional Neural Networks](https://mlsanity.com/paper/1708.04485/) (2017)
- [Accelerated Sparse Neural Training: A Provable and Efficient Method to Find N:M Transposable Masks](https://mlsanity.com/paper/2102.08124/) (2021)
- [Pixelated Butterfly: Simple and Efficient Sparse training for Neural Network Models](https://mlsanity.com/paper/2112.00029/) (2021)
- [SparseTIR: Composable Abstractions for Sparse Compilation in Deep Learning](https://mlsanity.com/paper/2207.04606/) (2022)
- [Block-Sparse Recurrent Neural Networks](https://mlsanity.com/paper/1711.02782/) (2017)
- [Compressing DMA Engine: Leveraging Activation Sparsity for Training Deep Neural Networks](https://mlsanity.com/paper/1705.01626/) (2017)
- [Learning Best Combination for Efficient N:M Sparsity](https://mlsanity.com/paper/2206.06662/) (2022)
- [Dual-side Sparse Tensor Core](https://mlsanity.com/paper/2105.09564/) (2021)
- [An Algorithm-Hardware Co-Optimized Framework for Accelerating N:M Sparse Transformers](https://mlsanity.com/paper/2208.06118/) (2022)
- [CoSA: Scheduling by Constrained Optimization for Spatial Accelerators](https://mlsanity.com/paper/2105.01898/) (2021)
- [S2TA: Exploiting Structured Sparsity for Energy-Efficient Mobile CNN Acceleration](https://mlsanity.com/paper/2107.07983/) (2021)
- [Progressive Gradient Flow for Robust N:M Sparsity Training in Transformers](https://mlsanity.com/paper/2402.04744/) (2024)
- [Systolic Tensor Array: An Efficient Structured-Sparse GEMM Accelerator for Mobile CNN Inference](https://mlsanity.com/paper/2005.08098/) (2020)
- [Sparseloop: An Analytical Approach To Sparse Tensor Accelerator Modeling](https://mlsanity.com/paper/2205.05826/) (2022)

## Built on by (most cited first, 8 of 8)

- [Demystifying AI Platform Design for Distributed Inference of Next-Generation LLM models](https://mlsanity.com/paper/2406.01698/) (2024)
- [Accelerating Transformer Inference and Training with 2:4 Activation Sparsity](https://mlsanity.com/paper/2503.16672/) (2025)
- [Scaling Down, Serving Fast: Compressing and Deploying Efficient LLMs for Recommendation Systems](https://mlsanity.com/paper/2502.14305/) (2025)
- [Different Prompts, Different Ranks: Prompt-aware Dynamic Rank Selection for SVD-based LLM Compression](https://mlsanity.com/paper/2605.08568/) (2026)
- [Leveraging Label Potential for Enhanced Multimodal Emotion Recognition](https://mlsanity.com/paper/2504.05158/) (2025)
- [TriADA: Massively Parallel Trilinear Matrix-by-Tensor Multiply-Add Algorithm and Device Architecture for the Acceleration of 3D Discrete Transformations](https://mlsanity.com/paper/2506.22818/) (2025)
- [To 2:4 Sparsity and Beyond: Neuron-level Activation Function to Accelerate LLM Pre-Training](https://mlsanity.com/paper/2602.06183/) (2026)
- [HieraSparse: Hierarchical Semi-Structured Sparse KV Attention](https://mlsanity.com/paper/2604.16864/) (2026)

## Abstract (arXiv)

Exploiting sparsity in deep neural networks (DNNs) has been a promising area for meeting the growing computation requirements. To minimize the overhead of sparse acceleration, hardware designers have proposed structured sparsity support, but it provides limited flexibility and requires extra model fine-tuning. Moreover, any sparse model fine-tuned for certain structured sparse HW cannot be accelerated by other structured hardware. To enable acceleration using unstructured sparsity of DNNs on structured sparse hardware, we propose an approximation method leveraging the distributive property in linear algebra to turn any sparse tensor into a series of structured sparse tensors. We also develop a software framework, TASDER, to apply high-quality structured approximation on weights and activations of DNNs. Our method accelerates dense and sparse DNNs without fine-tuning and improves energy-delay-product (EDP) by up to 83% and 74%. It achieves up to 39% speed-up on a real system.