# SystolicAttention: Fusing FlashAttention within a Single Systolic Array

arXiv: https://arxiv.org/abs/2507.11331 · HTML: https://mlsanity.com/paper/2507.11331/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 59 references that are in the map.
- First posted on arXiv 15 Jul 2025.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [In-Datacenter Performance Analysis of a Tensor Processing Unit](https://mlsanity.com/paper/1704.04760/) (2017)
- [Splitwise: Efficient generative LLM inference using phase splitting](https://mlsanity.com/paper/2311.18677/) (2023)
- [FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precision](https://mlsanity.com/paper/2407.08608/) (2024)
- [TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings](https://mlsanity.com/paper/2304.01433/) (2023)
- [FLAT: An Optimized Dataflow for Mitigating Attention Bottlenecks](https://mlsanity.com/paper/2107.06419/) (2021)
- [Interstellar: Using Halide's Scheduling Language to Analyze DNN Accelerators](https://mlsanity.com/paper/1809.04070/) (2018)
- [FuseMax: Leveraging Extended Einsums to Optimize Attention Accelerator Design](https://mlsanity.com/paper/2406.10491/) (2024)
- [DiP: A Scalable, Energy-Efficient Systolic Array for Matrix Multiplication Acceleration](https://mlsanity.com/paper/2412.09709/) (2024)
- [Low-Cost FlashAttention with Fused Exponential and Multiplication Hardware Operators](https://mlsanity.com/paper/2505.14314/) (2025)
- [TeAAL: A Declarative Framework for Modeling Sparse Tensor Accelerators](https://mlsanity.com/paper/2304.07931/) (2023)
- [A High-performance, Energy-efficient Modular DMA Engine Architecture](https://mlsanity.com/paper/2305.05240/) (2023)

## Built on by (most cited first, 3 of 3)

- [VersaQ-3D: Architecture Support for Visual Geometry Grounded Transformers via Versatile Quantization](https://mlsanity.com/paper/2601.20317/) (2026)
- [VitaLLM: A Versatile and Tiny Accelerator for Mixed-Precision LLM Inference on Edge Devices](https://mlsanity.com/paper/2605.00320/) (2026)
- [From Pragmas to Partners: A Symbiotic Evolution of Agentic High-Level Synthesis](https://mlsanity.com/paper/2602.01401/) (2026)

## Abstract (arXiv)

Transformer models rely heavily on the scaled dot-product attention (SDPA) operation, typically implemented as FlashAttention. Characterized by its frequent interleaving of matrix multiplications and softmax operations, FlashAttention fails to fully utilize the compute resources of modern systolic-array-based accelerators designed for consecutive and large matrix multiplications. To fully unleash the performance potential of systolic arrays for FlashAttention, we propose FSA, an enhanced systolic array architecture that runs the entire FlashAttention on the array without external vector units. Combined with SystolicAttention, an optimized kernel for FSA that achieves fine-grained and element-wise overlapping of FlashAttention operations, FSA maximizes array utilization while preserving the original floating-point operation order of FlashAttention. We implement FSA in synthesizable RTL and evaluate its performance against state-of-the-art systolic-array-based accelerators. Our results show that FSA achieves 1.77x and 4.83x higher attention FLOPs/s utilization compared to AWS Neuron-v2 and Google TPUv5e, respectively. We synthesize FSA in a 16 nm technology at 1.5 GHz, and results indicate only a 12% area overhead compared to a standard weight-stationary systolic array.