# Systolic Tensor Array: An Efficient Structured-Sparse GEMM Accelerator for Mobile CNN Inference

arXiv: https://arxiv.org/abs/2005.08098 · HTML: https://mlsanity.com/paper/2005.08098/

## Key facts

- Cited by 17 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 May 2020.

## Built on by (most cited first, 17 of 17)

- [An Algorithm-Hardware Co-Optimized Framework for Accelerating N:M Sparse Transformers](https://mlsanity.com/paper/2208.06118/) (2022)
- [S2TA: Exploiting Structured Sparsity for Energy-Efficient Mobile CNN Acceleration](https://mlsanity.com/paper/2107.07983/) (2021)
- [Dynamic N:M Fine-grained Structured Sparse Attention Mechanism](https://mlsanity.com/paper/2203.00091/) (2022)
- [VEGETA: Vertically-Integrated Extensions for Sparse/Dense GEMM Tile Acceleration on CPUs](https://mlsanity.com/paper/2302.08687/) (2023)
- [Enabling Unstructured Sparse Acceleration on Structured Sparse Accelerators](https://mlsanity.com/paper/2403.07953/) (2024)
- [Doping: A technique for efficient compression of LSTM models using sparse structured additive matrices](https://mlsanity.com/paper/2102.07071/) (2021)
- [Scale-out Systolic Arrays](https://mlsanity.com/paper/2203.11540/) (2022)
- [SMASH: Sparse Matrix Atomic Scratchpad Hashing](https://mlsanity.com/paper/2105.14156/) (2021)
- [Integration of a systolic array based hardware accelerator into a DNN operator auto-tuning framework](https://mlsanity.com/paper/2212.03034/) (2022)
- [MVQ:Towards Efficient DNN Compression and Acceleration with Masked Vector Quantization](https://mlsanity.com/paper/2412.10261/) (2024)
- [tuGEMM: Area-Power-Efficient Temporal Unary GEMM Architecture for Low-Precision Edge AI](https://mlsanity.com/paper/2412.17966/) (2024)
- [Systolic Arrays and Structured Pruning Co-design for Efficient Transformers in Edge Systems](https://mlsanity.com/paper/2411.10285/) (2024)
- [Periodic Online Testing for Sparse Systolic Tensor Arrays](https://mlsanity.com/paper/2504.18628/) (2025)
- [A Construction Kit for Efficient Low Power Neural Network Accelerator Designs](https://mlsanity.com/paper/2106.12810/) (2021)
- [CORDIC Is All You Need](https://mlsanity.com/paper/2503.11685/) (2025)
- [MX-SAFE: Versatile Inference- and Training-Proof Microscaling Format with On-the-Fly Exponent and Mantissa Bit Allocation](https://mlsanity.com/paper/2605.24391/) (2026)
- [Precision-Aware Variable Bit Processing Elements for Hardware-Efficient Systolic Array Designs](https://mlsanity.com/paper/2608.22378/) (2026)

## Abstract (arXiv)

Convolutional neural network (CNN) inference on mobile devices demands efficient hardware acceleration of low-precision (INT8) general matrix multiplication (GEMM). The systolic array (SA) is a pipelined 2D array of processing elements (PEs), with very efficient local data movement, well suited to accelerating GEMM, and widely deployed in industry. In this work, we describe two significant improvements to the traditional SA architecture, to specifically optimize for CNN inference. Firstly, we generalize the traditional scalar PE, into a Tensor-PE, which gives rise to a family of new Systolic Tensor Array (STA) microarchitectures. The STA family increases intra-PE operand reuse and datapath efficiency, resulting in circuit area and power dissipation reduction of as much as 2.08x and 1.36x respectively, compared to the conventional SA at iso-throughput with INT8 operands. Secondly, we extend this design to support a novel block-sparse data format called density-bound block (DBB). This variant (STA-DBB) achieves a 3.14x and 1.97x improvement over the SA baseline at iso-throughput in area and power respectively, when processing specially-trained DBB-sparse models, while remaining fully backwards compatible with dense models.