# In Search of Needles in a 11M Haystack: Recurrent Memory Finds What LLMs Miss

arXiv: https://arxiv.org/abs/2402.10790 · HTML: https://mlsanity.com/paper/2402.10790/

## Key facts

- Cited by 36 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 39 references that are in the map.
- First posted on arXiv 16 Feb 2024.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Mamba: Linear-Time Sequence Modeling with Selective State Spaces](https://mlsanity.com/paper/2312.00752/) (2023)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [REALM: Retrieval-Augmented Language Model Pre-Training](https://mlsanity.com/paper/2002.08909/) (2020)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding](https://mlsanity.com/paper/2308.14508/) (2023)
- [Improving language models by retrieving from trillions of tokens](https://mlsanity.com/paper/2112.04426/) (2021)
- [RWKV: Reinventing RNNs for the Transformer Era](https://mlsanity.com/paper/2305.13048/) (2023)
- [Generalization through Memorization: Nearest Neighbor Language Models](https://mlsanity.com/paper/1911.00172/) (2019)
- [REPLUG: Retrieval-Augmented Black-Box Language Models](https://mlsanity.com/paper/2301.12652/) (2023)
- [Compressive Transformers for Long-Range Sequence Modelling](https://mlsanity.com/paper/1911.05507/) (2019)
- [Long Range Arena: A Benchmark for Efficient Transformers](https://mlsanity.com/paper/2011.04006/) (2020)
- [Neural Turing Machines](https://mlsanity.com/paper/1410.5401/) (2014)
- [Adapting Language Models to Compress Contexts](https://mlsanity.com/paper/2305.14788/) (2023)
- [NLP Evaluation in trouble: On the Need to Measure LLM Data Contamination for each Benchmark](https://mlsanity.com/paper/2310.18018/) (2023)
- [Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks](https://mlsanity.com/paper/1502.05698/) (2015)
- [L-Eval: Instituting Standardized Evaluation for Long Context Language Models](https://mlsanity.com/paper/2307.11088/) (2023)
- [Memorizing Transformers](https://mlsanity.com/paper/2203.08913/) (2022)
- [LongNet: Scaling Transformers to 1,000,000,000 Tokens](https://mlsanity.com/paper/2307.02486/) (2023)
- [Recurrent Memory Transformer](https://mlsanity.com/paper/2207.06881/) (2022)
- [ZeroSCROLLS: A Zero-Shot Benchmark for Long Text Understanding](https://mlsanity.com/paper/2305.14196/) (2023)
- [Block-Recurrent Transformers](https://mlsanity.com/paper/2203.07852/) (2022)
- [Long Context Compression with Activation Beacon](https://mlsanity.com/paper/2401.03462/) (2024)
- [Scaling Transformer to 1M tokens and beyond with RMT](https://mlsanity.com/paper/2304.11062/) (2023)
- [MART: Memory-Augmented Recurrent Transformer for Coherent Video Paragraph Captioning](https://mlsanity.com/paper/2005.05402/) (2020)
- [Memformer: A Memory-Augmented Transformer for Sequence Modeling](https://mlsanity.com/paper/2010.06891/) (2020)
- [Retrieval-Pretrained Transformer: Long-range Language Modeling with Self-retrieval](https://mlsanity.com/paper/2306.13421/) (2023)
- [Temporal Latent Bottleneck: Synthesis of Fast and Slow Processing Mechanisms in Sequence Learning](https://mlsanity.com/paper/2205.14794/) (2022)

## Built on by (most cited first, 30 of 36)

- [From Local to Global: A Graph RAG Approach to Query-Focused Summarization](https://mlsanity.com/paper/2404.16130/) (2024)
- [Retrieval-Augmented Generation for AI-Generated Content: A Survey](https://mlsanity.com/paper/2402.19473/) (2024)
- [Retrieval Head Mechanistically Explains Long-Context Factuality](https://mlsanity.com/paper/2404.15574/) (2024)
- [Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scale](https://mlsanity.com/paper/2504.14225/) (2025)
- [Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption](https://mlsanity.com/paper/2407.18003/) (2024)
- [MileBench: Benchmarking MLLMs in Long Context](https://mlsanity.com/paper/2404.18532/) (2024)
- [From Isolated Conversations to Hierarchical Schemas: Dynamic Tree Memory Representation for LLMs](https://mlsanity.com/paper/2410.14052/) (2024)
- [RNNs are not Transformers (Yet): The Key Bottleneck on In-context Retrieval](https://mlsanity.com/paper/2402.18510/) (2024)
- [Needle In A Multimodal Haystack](https://mlsanity.com/paper/2406.07230/) (2024)
- [Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Models](https://mlsanity.com/paper/2406.11230/) (2024)
- [SampleAttention: Near-Lossless Acceleration of Long Context LLM Inference with Adaptive Structured Sparse Attention](https://mlsanity.com/paper/2406.15486/) (2024)
- [Needle in the Haystack for Memory Based Large Language Models](https://mlsanity.com/paper/2407.01437/) (2024)
- [TRIX: A More Expressive Model for Zero-shot Domain Transfer in Knowledge Graphs](https://mlsanity.com/paper/2502.19512/) (2025)
- [DetectiveQA: Evaluating Long-Context Reasoning on Detective Novels](https://mlsanity.com/paper/2409.02465/) (2024)
- [VideoLLaMB: Long Streaming Video Understanding with Recurrent Memory Bridges](https://mlsanity.com/paper/2409.01071/) (2024)
- [Associative Recurrent Memory Transformer](https://mlsanity.com/paper/2407.04841/) (2024)
- [Needle Threading: Can LLMs Follow Threads through Near-Million-Scale Haystacks?](https://mlsanity.com/paper/2411.05000/) (2024)
- [Context Is What You Need: The Maximum Effective Context Window for Real World Limits of LLMs](https://mlsanity.com/paper/2509.21361/) (2025)
- [VTCBench: Can Vision-Language Models Understand Long Context with Vision-Text Compression?](https://mlsanity.com/paper/2512.15649/) (2025)
- [Bug In the Code Stack: Can LLMs Find Bugs in Large Python Code Stacks](https://mlsanity.com/paper/2406.15325/) (2024)
- [ActMem: Bridging the Gap Between Memory Retrieval and Reasoning in LLM Agents](https://mlsanity.com/paper/2603.00026/) (2026)
- [Enhancing Supermarket Robot Interaction: A Multi-Level LLM Conversational Interface for Handling Diverse Customer Intents](https://mlsanity.com/paper/2406.11047/) (2024)
- [What to Keep, What to Forget: A Rate--Distortion View of Memory Compaction in LLMs and Agents](https://mlsanity.com/paper/2607.08032/) (2026)
- [SolidMark: Evaluating Image Memorization in Generative Models](https://mlsanity.com/paper/2503.00592/) (2025)
- [JobHop: A Large-Scale Dataset of Career Trajectories](https://mlsanity.com/paper/2505.07653/) (2025)
- [NeMo: Needle in a Montage for Video-Language Understanding](https://mlsanity.com/paper/2509.24563/) (2025)
- [Assessing the quality of information extraction](https://mlsanity.com/paper/2404.04068/) (2024)
- [Core-based Hierarchies for Efficient GraphRAG](https://mlsanity.com/paper/2603.05207/) (2026)
- [Microstructures and Accuracy of Graph Recall by Large Language Models](https://mlsanity.com/paper/2402.11821/) (2024)
- [Pay Attention to What Matters](https://mlsanity.com/paper/2409.19001/) (2024)

## Abstract (arXiv)

This paper addresses the challenge of processing long documents using generative transformer models. To evaluate different approaches, we introduce BABILong, a new benchmark designed to assess model capabilities in extracting and processing distributed facts within extensive texts. Our evaluation, which includes benchmarks for GPT-4 and RAG, reveals that common methods are effective only for sequences up to 104 elements. In contrast, fine-tuning GPT-2 with recurrent memory augmentations enables it to handle tasks involving up to 11× 106 elements. This achievement marks a substantial leap, as it is by far the longest input processed by any neural network model to date, demonstrating a significant improvement in the processing capabilities for long sequences.