# Pay Attention when Required

arXiv: https://arxiv.org/abs/2009.04534 · HTML: https://mlsanity.com/paper/2009.04534/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 9 Sep 2020.

## Built on by (most cited first, 8 of 8)

- [Generating Images with Sparse Representations](https://mlsanity.com/paper/2103.03841/) (2021)
- [Understanding Robustness of Transformers for Image Classification](https://mlsanity.com/paper/2103.14586/) (2021)
- [Finetuning Pretrained Transformers into RNNs](https://mlsanity.com/paper/2103.13076/) (2021)
- [Attend First, Consolidate Later: On the Importance of Attention in Different LLM Layers](https://mlsanity.com/paper/2409.03621/) (2024)
- [Revisiting Simple Neural Probabilistic Language Models](https://mlsanity.com/paper/2104.03474/) (2021)
- [Can the Transformer Be Used as a Drop-in Replacement for RNNs in Text-Generating GANs?](https://mlsanity.com/paper/2108.12275/) (2021)
- [Efficient Sparsely Activated Transformers](https://mlsanity.com/paper/2208.14580/) (2022)
- [Efficient Language Modeling for Low-Resource Settings with Hybrid RNN-Transformer Architectures](https://mlsanity.com/paper/2502.00617/) (2025)

## Abstract (arXiv)

Transformer-based models consist of interleaved feed-forward blocks - that capture content meaning, and relatively more expensive self-attention blocks - that capture context meaning. In this paper, we explored trade-offs and ordering of the blocks to improve upon the current Transformer architecture and proposed PAR Transformer. It needs 35% lower compute time than Transformer-XL achieved by replacing ~63% of the self-attention blocks with feed-forward blocks, and retains the perplexity on WikiText-103 language modelling benchmark. We further validated our results on text8 and enwiki8 datasets, as well as on the BERT model.