# Frustratingly Short Attention Spans in Neural Language Modeling

arXiv: https://arxiv.org/abs/1702.04521 · HTML: https://mlsanity.com/paper/1702.04521/

## Key facts

- Cited by 18 papers in the map (data through 7 Oct 2026).
- First posted on arXiv 15 Feb 2017.

## Built on by (most cited first, 18 of 18)

- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [A General Survey on Attention Mechanisms in Deep Learning](https://mlsanity.com/paper/2203.14263/) (2022)
- [Attention, please! A survey of Neural Attention Models in Deep Learning](https://mlsanity.com/paper/2103.16775/) (2021)
- [Looking Beyond Text: Reducing Language bias in Large Vision-Language Models via Multimodal Dual-Attention and Soft-Image Guidance](https://mlsanity.com/paper/2411.14279/) (2024)
- [Artificial Intelligence for the Metaverse: A Survey](https://mlsanity.com/paper/2202.10336/) (2022)
- [Memory-guided Unsupervised Image-to-image Translation](https://mlsanity.com/paper/2104.05170/) (2021)
- [The Health Gym: Synthetic Health-Related Datasets for the Development of Reinforcement Learning Algorithms](https://mlsanity.com/paper/2203.06369/) (2022)
- [ReelFramer: Human-AI Co-Creation for News-to-Video Translation](https://mlsanity.com/paper/2304.09653/) (2023)
- [A Survey on Aspect-Based Sentiment Classification](https://mlsanity.com/paper/2203.14266/) (2022)
- [In-Context Exemplars as Clues to Retrieving from Large Associative Memory](https://mlsanity.com/paper/2311.03498/) (2023)
- [Embedding Graph Convolutional Networks in Recurrent Neural Networks for Predictive Monitoring](https://mlsanity.com/paper/2112.09641/) (2021)
- [LLM-TAKE: Theme Aware Keyword Extraction Using Large Language Models](https://mlsanity.com/paper/2312.00909/) (2023)
- [Zero-Shot Controlled Generation with Encoder-Decoder Transformers](https://mlsanity.com/paper/2106.06411/) (2021)
- [Prompt-Aware Adapter: Towards Learning Adaptive Visual Tokens for Multimodal Large Language Models](https://mlsanity.com/paper/2405.15684/) (2024)
- [State-Regularized Recurrent Neural Networks to Extract Automata and Explain Predictions](https://mlsanity.com/paper/2212.05178/) (2022)
- [Learning a Word-Level Language Model with Sentence-Level Noise Contrastive Estimation for Contextual Sentence Probability Estimation](https://mlsanity.com/paper/2103.07875/) (2021)
- [Measuring Linguistic Diversity During COVID-19](https://mlsanity.com/paper/2104.01290/) (2021)
- [Pruning Literals for Highly Efficient Explainability at Word Level](https://mlsanity.com/paper/2411.04557/) (2024)

## Abstract (arXiv)

Neural language models predict the next token using a latent representation of the immediate token history. Recently, various methods for augmenting neural language models with an attention mechanism over a differentiable memory have been proposed. For predicting the next token, these models query information from a memory of the recent history which can facilitate learning mid- and long-range dependencies. However, conventional attention mechanisms used in memory-augmented neural language models produce a single output vector per time step. This vector is used both for predicting the next token as well as for the key and value of a differentiable memory of a token history. In this paper, we propose a neural language model with a key-value attention mechanism that outputs separate representations for the key and value of a differentiable memory, as well as for encoding the next-word distribution. This model outperforms existing memory-augmented neural language models on two corpora. Yet, we found that our method mainly utilizes a memory of the five most recent output representations. This led to the unexpected main finding that a much simpler model based only on the concatenation of recent output representations from previous time steps is on par with more sophisticated memory-augmented neural language models.