# Attention Overflow: Language Model Input Blur during Long-Context Missing Items Recommendation

arXiv: https://arxiv.org/abs/2407.13481 · HTML: https://mlsanity.com/paper/2407.13481/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 15 references that are in the map.
- First posted on arXiv 18 Jul 2024.

## Builds on (most cited first, 12 of 12)

- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://mlsanity.com/paper/1909.05858/) (2019)
- [Long Range Arena: A Benchmark for Efficient Transformers](https://mlsanity.com/paper/2011.04006/) (2020)
- [A Survey on Large Language Models for Recommendation](https://mlsanity.com/paper/2305.19860/) (2023)
- [Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks](https://mlsanity.com/paper/1502.05698/) (2015)
- [Transformers as Soft Reasoners over Language](https://mlsanity.com/paper/2002.05867/) (2020)
- [BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack](https://mlsanity.com/paper/2406.10149/) (2024)
- [A Theoretical Analysis of the Repetition Problem in Text Generation](https://mlsanity.com/paper/2012.14660/) (2020)
- [Zero-Shot Recommendation as Language Modeling](https://mlsanity.com/paper/2112.04184/) (2021)
- [Over-Reasoning and Redundant Calculation of Large Language Models](https://mlsanity.com/paper/2401.11467/) (2024)
- [Extending Llama-3's Context Ten-Fold Overnight](https://mlsanity.com/paper/2404.19553/) (2024)

## Built on by (most cited first, 2 of 2)

- [Reasoning Core: A Scalable RL Environment for LLM Symbolic Reasoning](https://mlsanity.com/paper/2509.18083/) (2025)
- [AI5GTest: AI-Driven Specification-Aware Automated Testing and Validation of 5G O-RAN Components](https://mlsanity.com/paper/2506.10111/) (2025)

## Abstract (arXiv)

Large language models (LLMs) can suggest missing elements from items listed in a prompt, which can be used for list completion or recommendations based on users' history. However, their performance degrades when presented with too many items, as they start to suggest items already included in the input list. This occurs at around 100 items for mid-2024 flagship LLMs. We evaluate this phenomenon on both synthetic problems (e.g., finding missing numbers in a given range of shuffled integers) and realistic movie recommendation scenarios. We refer to this issue as textitattention overflow, as preventing repetition requires attending to all items simultaneously. Although iterative loops can mitigate this problem, their costs increase with the repetition rate, affecting the language models' ability to derive novelty from lengthy inputs.