# SISA: A Scale-In Systolic Array for GEMM Acceleration

arXiv: https://arxiv.org/abs/2603.29913 · HTML: https://mlsanity.com/paper/2603.29913/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 21 references that are in the map.
- First posted on arXiv 31 Mar 2026.

## Builds on (most cited first, 10 of 10)

- [Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve](https://mlsanity.com/paper/2403.02310/) (2024)
- [DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scale](https://mlsanity.com/paper/2207.00032/) (2022)
- [Gemmini: Enabling Systematic Deep-Learning Architecture Evaluation via Full-Stack Integration](https://mlsanity.com/paper/1911.09925/) (2019)
- [Google Neural Network Models for Edge Devices: Analyzing and Mitigating Machine Learning Inference Bottlenecks](https://mlsanity.com/paper/2109.14320/) (2021)
- [SCALE-Sim v3: A modular cycle-accurate systolic accelerator simulator for end-to-end system analysis](https://mlsanity.com/paper/2504.15377/) (2025)
- [LLM Serving Optimization with Variable Prefill and Decode Lengths](https://mlsanity.com/paper/2508.06133/) (2025)
- [FlexSA: Flexible Systolic Array Architecture for Efficient Pruned DNN Model Training](https://mlsanity.com/paper/2004.13027/) (2020)
- [Scale-out Systolic Arrays](https://mlsanity.com/paper/2203.11540/) (2022)
- [ReGate: Enabling Power Gating in Neural Processing Units](https://mlsanity.com/paper/2508.02536/) (2025)
- [ReDas: A Lightweight Architecture for Supporting Fine-Grained Reshaping and Multiple Dataflows on Systolic Array](https://mlsanity.com/paper/2302.07520/) (2023)

## Abstract (arXiv)

The currently dominant AI/ML workloads, such as Large Language Models (LLMs), rely on the efficient execution of General Matrix-Matrix Multiplication (GEMM) operations. Thus, most systems are equipped with dedicated matrix hardware accelerators based on square Systolic Arrays (SAs) of Processing Elements (PEs). While this organization was effective for traditional Deep Neural Networks (DNNs), LLMs introduce input-dependent and highly skewed matrices, leading to underutilized SA resources. To address this challenge, we propose SISA (Scale-In Systolic Array), a novel SA architecture that partitions the traditional square array into horizontal rectangular slabs. With minimal overhead, SISA exposes parallelism through independently scheduled slabs for efficient execution of small or skewed matrix shapes, while retaining full-array operation for large GEMMs. SISA achieves up to 8.52x speedup and 93% energy-delay-product (EDP) reduction for representative LLMs compared to a state-of-the-art monolithic SA with the same number of PEs.