# Needle in the Haystack for Memory Based Large Language Models

arXiv: https://arxiv.org/abs/2407.01437 · HTML: https://mlsanity.com/paper/2407.01437/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 16 references that are in the map.
- First posted on arXiv 1 Jul 2024.

## Builds on (most cited first, 11 of 11)

- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU](https://mlsanity.com/paper/2303.06865/) (2023)
- [Landmark Attention: Random-Access Infinite Context Length for Transformers](https://mlsanity.com/paper/2305.16300/) (2023)
- [Leave No Context Behind: Efficient Infinite Context Transformers with Infini-attention](https://mlsanity.com/paper/2404.07143/) (2024)
- [Optimus: Organizing Sentences via Pre-trained Modeling of a Latent Space](https://mlsanity.com/paper/2004.04092/) (2020)
- [Rewriting a Deep Generative Model](https://mlsanity.com/paper/2007.15646/) (2020)
- [Memory Transformer](https://mlsanity.com/paper/2006.11527/) (2020)
- [Larimar: Large Language Models with Episodic Memory Control](https://mlsanity.com/paper/2403.11901/) (2024)
- [In Search of Needles in a 11M Haystack: Recurrent Memory Finds What LLMs Miss](https://mlsanity.com/paper/2402.10790/) (2024)
- [TransformerFAM: Feedback attention is working memory](https://mlsanity.com/paper/2404.09173/) (2024)

## Built on by (most cited first, 30 of 34)

- [Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory](https://mlsanity.com/paper/2504.19413/) (2025)
- [Which Agent Causes Task Failures and When? On Automated Failure Attribution of LLM Multi-Agent Systems](https://mlsanity.com/paper/2505.00212/) (2025)
- [Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scale](https://mlsanity.com/paper/2504.14225/) (2025)
- [T*: Re-thinking Temporal Search for Long-Form Video Understanding](https://mlsanity.com/paper/2504.02259/) (2025)
- [LLMs Will Always Hallucinate, and We Need to Live With This](https://mlsanity.com/paper/2409.05746/) (2024)
- [AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications](https://mlsanity.com/paper/2602.22769/) (2026)
- [Beyond the Final Answer: Evaluating the Reasoning Trajectories of Tool-Augmented Agents](https://mlsanity.com/paper/2510.02837/) (2025)
- [Evaluating Long-Horizon Memory for Multi-Party Collaborative Dialogues](https://mlsanity.com/paper/2602.01313/) (2026)
- [Distance between Relevant Information Pieces Causes Bias in Long-Context LLMs](https://mlsanity.com/paper/2410.14641/) (2024)
- [Context Is What You Need: The Maximum Effective Context Window for Real World Limits of LLMs](https://mlsanity.com/paper/2509.21361/) (2025)
- [BLAB: Brutally Long Audio Bench](https://mlsanity.com/paper/2505.03054/) (2025)
- [The Lucie-7B LLM and the Lucie Training Dataset: Open resources for multilingual language generation](https://mlsanity.com/paper/2503.12294/) (2025)
- [Tool-Augmented Spatiotemporal Reasoning for Streamlining Video Question Answering Task](https://mlsanity.com/paper/2512.10359/) (2025)
- [Towards Natural Personalization: Evaluating Long-Horizon Preference Following in Personalized User-LLM Interactions](https://mlsanity.com/paper/2603.04191/) (2026)
- [Evaluating the Sensitivity of LLMs to Prior Context](https://mlsanity.com/paper/2506.00069/) (2025)
- [Prompt-MII: Meta-Learning Instruction Induction for LLMs](https://mlsanity.com/paper/2510.16932/) (2025)
- [From Tools to Teammates: Evaluating LLMs in Multi-Session Coding Interactions](https://mlsanity.com/paper/2502.13791/) (2025)
- [M2RNN: Non-Linear RNNs with Matrix-Valued States for Scalable Language Modeling](https://mlsanity.com/paper/2603.14360/) (2026)
- [Video-LevelGauge: Investigating Contextual Positional Bias in Large Video Language Models](https://mlsanity.com/paper/2508.19650/) (2025)
- [Mi:dm 2.0 Korea-centric Bilingual Language Models](https://mlsanity.com/paper/2601.09066/) (2026)
- [K2-V2: A 360-Open, Reasoning-Enhanced LLM](https://mlsanity.com/paper/2512.06201/) (2025)
- [SCOPE: Stochastic and Counterbiased Option Placement for Evaluating Large Language Models](https://mlsanity.com/paper/2507.18182/) (2025)
- [Mask-Enhanced Autoregressive Prediction: Pay Less Attention to Learn More](https://mlsanity.com/paper/2502.07490/) (2025)
- [Pause-Tuning for Long-Context Comprehension: A Lightweight Approach to LLM Attention Recalibration](https://mlsanity.com/paper/2502.20405/) (2025)
- [PromptDistill: Query-based Selective Token Retention in Intermediate Layers for Efficient Large Language Model Inference](https://mlsanity.com/paper/2503.23274/) (2025)
- [Can an Actor-Critic Optimization Framework Improve Analog Design?](https://mlsanity.com/paper/2603.24714/) (2026)
- [From Facts to Insights: A Persona-Driven Dual Memory Framework and Dataset for Role-Playing Agents](https://mlsanity.com/paper/2605.25693/) (2026)
- [Assessing Consciousness-Related Behaviors in Large Language Models Using the Maze Test](https://mlsanity.com/paper/2508.16705/) (2025)
- [Active Learning and Explainable AI for Multi-Objective Optimization of Spin Coated Polymers](https://mlsanity.com/paper/2509.08988/) (2025)
- [CareMedEval dataset: Evaluating Critical Appraisal and Reasoning in the Biomedical Field](https://mlsanity.com/paper/2511.03441/) (2025)

## Abstract (arXiv)

Current large language models (LLMs) often perform poorly on simple fact retrieval tasks. Here we investigate if coupling a dynamically adaptable external memory to a LLM can alleviate this problem. For this purpose, we test Larimar, a recently proposed language model architecture which uses an external associative memory, on long-context recall tasks including passkey and needle-in-the-haystack tests. We demonstrate that the external memory of Larimar, which allows fast write and read of an episode of text samples, can be used at test time to handle contexts much longer than those seen during training. We further show that the latent readouts from the memory (to which long contexts are written) control the decoder towards generating correct outputs, with the memory stored off of the GPU. Compared to existing transformer-based LLM architectures for long-context recall tasks that use larger parameter counts or modified attention mechanisms, a relatively smaller size Larimar is able to maintain strong performance without any task-specific training or training on longer contexts.