# Can Active Memory Replace Attention?

arXiv: https://arxiv.org/abs/1610.08613 · HTML: https://mlsanity.com/paper/1610.08613/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Oct 2016.

## Built on by (most cited first, 6 of 6)

- [A Complete Survey on Generative AI (AIGC): Is ChatGPT from GPT-4 to GPT-5 All You Need?](https://mlsanity.com/paper/2303.11717/) (2023)
- [Universal Spam Detection using Transfer Learning of BERT Model](https://mlsanity.com/paper/2202.03480/) (2022)
- [In-Context Exemplars as Clues to Retrieving from Large Associative Memory](https://mlsanity.com/paper/2311.03498/) (2023)
- [Large Language Models are In-context Teachers for Knowledge Reasoning](https://mlsanity.com/paper/2311.06985/) (2023)
- [TCNet: Continuous Sign Language Recognition from Trajectories and Correlated Regions](https://mlsanity.com/paper/2403.11818/) (2024)
- [A Multi-Layer Regression based Predicable Function Fitting Network](https://mlsanity.com/paper/2209.09647/) (2022)

## Abstract (arXiv)

Several mechanisms to focus attention of a neural network on selected parts of its input or memory have been used successfully in deep learning models in recent years. Attention has improved image classification, image captioning, speech recognition, generative models, and learning algorithmic tasks, but it had probably the largest impact on neural machine translation. Recently, similar improvements have been obtained using alternative mechanisms that do not focus on a single part of a memory but operate on all of it in parallel, in a uniform way. Such mechanism, which we call active memory, improved over attention in algorithmic tasks, image processing, and in generative modelling. So far, however, active memory has not improved over attention for most natural language processing tasks, in particular for machine translation. We analyze this shortcoming in this paper and propose an extended model of active memory that matches existing attention models on neural machine translation and generalizes better to longer sentences. We investigate this model and explain why previous active memory models did not succeed. Finally, we discuss when active memory brings most benefits and where attention can be a better choice.