# An Efficient Data Reuse with Tile-Based Adaptive Stationary for Transformer Accelerators

arXiv: https://arxiv.org/abs/2503.19640 · HTML: https://mlsanity.com/paper/2503.19640/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 7 of its 10 references that are in the map.
- First posted on arXiv 25 Mar 2025.

## Builds on (most cited first, 7 of 7)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [XLS-R: Self-supervised Cross-lingual Speech Representation Learning at Scale](https://mlsanity.com/paper/2111.09296/) (2021)

## Abstract (arXiv)

Transformer-based models have become the textitde facto backbone across many fields, such as computer vision and natural language processing. However, as these models scale in size, external memory access (EMA) for weight and activations becomes a critical bottleneck due to its significantly higher energy consumption compared to internal computations. While most prior work has focused on optimizing the self-attention mechanism, little attention has been given to optimizing data transfer during linear projections, where EMA costs are equally important. In this paper, we propose the Tile-based Adaptive Stationary (TAS) scheme that selects the input or weight stationary in a tile granularity, based on the input sequence length. Our experimental results demonstrate that TAS can significantly reduce EMA by more than 97% compared to traditional stationary schemes, while being compatible with various attention optimization techniques and hardware accelerators.