# Is Attention All What You Need? -- An Empirical Investigation on Convolution-Based Active Memory and Self-Attention

arXiv: https://arxiv.org/abs/1912.11959 · HTML: https://mlsanity.com/paper/1912.11959/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Dec 2019.

## Built on by (most cited first, 1 of 1)

- [Attention Is Not All You Need Anymore](https://mlsanity.com/paper/2308.07661/) (2023)

## Abstract (arXiv)

The key to a Transformer model is the self-attention mechanism, which allows the model to analyze an entire sequence in a computationally efficient manner. Recent work has suggested the possibility that general attention mechanisms used by RNNs could be replaced by active-memory mechanisms. In this work, we evaluate whether various active-memory mechanisms could replace self-attention in a Transformer. Our experiments suggest that active-memory alone achieves comparable results to the self-attention mechanism for language modelling, but optimal results are mostly achieved by using both active-memory and self-attention mechanisms together. We also note that, for some specific algorithmic tasks, active-memory mechanisms alone outperform both self-attention and a combination of the two.