# Scale-out Systolic Arrays

arXiv: https://arxiv.org/abs/2203.11540 · HTML: https://mlsanity.com/paper/2203.11540/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 69 references that are in the map.
- First posted on arXiv 22 Mar 2022.

## Builds on (most cited first, 15 of 15)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Rethinking the Inception Architecture for Computer Vision](https://mlsanity.com/paper/1512.00567/) (2015)
- [In-Datacenter Performance Analysis of a Tensor Processing Unit](https://mlsanity.com/paper/1704.04760/) (2017)
- [Efficient Processing of Deep Neural Networks: A Tutorial and Survey](https://mlsanity.com/paper/1703.09039/) (2017)
- [EIE: Efficient Inference Engine on Compressed Deep Neural Network](https://mlsanity.com/paper/1602.01528/) (2016)
- [Bit Fusion: Bit-Level Dynamically Composable Architecture for Accelerating Deep Neural Networks](https://mlsanity.com/paper/1712.01507/) (2017)
- [SuperNeurons: Dynamic GPU Memory Management for Training Deep Neural Networks](https://mlsanity.com/paper/1801.04380/) (2018)
- [PREMA: A Predictive Multi-task Scheduling Algorithm For Preemptible Neural Processing Units](https://mlsanity.com/paper/1909.04548/) (2019)
- [SCALE-Sim: Systolic CNN Accelerator Simulator](https://mlsanity.com/paper/1811.02883/) (2018)
- [Systolic Tensor Array: An Efficient Structured-Sparse GEMM Accelerator for Mobile CNN Inference](https://mlsanity.com/paper/2005.08098/) (2020)
- [Packing Sparse Convolutional Neural Networks for Efficient Systolic Array Implementations: Column Combining Under Joint Optimization](https://mlsanity.com/paper/1811.04770/) (2018)
- [Caffeinated FPGAs: FPGA Framework For Convolutional Neural Networks](https://mlsanity.com/paper/1609.09671/) (2016)
- [Sparse Systolic Tensor Array for Efficient CNN Hardware Acceleration](https://mlsanity.com/paper/2009.02381/) (2020)

## Built on by (most cited first, 5 of 5)

- [TrIM, Triangular Input Movement Systolic Array for Convolutional Neural Networks: Architecture and Hardware Implementation](https://mlsanity.com/paper/2408.10243/) (2024)
- [Strassen Multisystolic Array Hardware Architectures](https://mlsanity.com/paper/2502.10063/) (2025)
- [Explainable AI-Guided Efficient Approximate DNN Generation for Multi-Pod Systolic Arrays](https://mlsanity.com/paper/2503.16583/) (2025)
- [D-Legion: A Scalable Many-Core Architecture for Accelerating Matrix Multiplication in Quantized LLMs](https://mlsanity.com/paper/2602.06252/) (2026)
- [SISA: A Scale-In Systolic Array for GEMM Acceleration](https://mlsanity.com/paper/2603.29913/) (2026)

## Abstract (arXiv)

Multi-pod systolic arrays are emerging as the architecture of choice in DNN inference accelerators. Despite their potential, designing multi-pod systolic arrays to maximize effective throughput/Watt (i.e., throughput/Watt adjusted when accounting for array utilization) poses a unique set of challenges. In this work, we study three key pillars in multi-pod systolic array designs, namely array granularity, interconnect, and tiling. We identify optimal array granularity across workloads and show that state-of-the-art commercial accelerators use suboptimal array sizes for single-tenancy workloads. We, then evaluate the bandwidth/latency trade-offs in interconnects and show that Butterfly networks offer a scalable topology for accelerators with a large number of pods. Finally, we introduce a novel data tiling scheme with custom partition size to maximize utilization in optimally sized pods. We propose Scale-out Systolic Arrays, a multi-pod inference accelerator for both single- and multi-tenancy based on these three pillars. We show that SOSA exhibits scaling of up to 600 TeraOps/s in effective throughput for state-of-the-art DNN inference workloads, and outperforms state-of-the-art multi-pod accelerators by a factor of 1.5x.