# Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMs

arXiv: https://arxiv.org/abs/2510.27246 · HTML: https://mlsanity.com/paper/2510.27246/

## Key facts

- Cited by 60 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 Oct 2025.

## Built on by (most cited first, 30 of 60)

- [Hindsight is 20/20: Building Agent Memory that Retains, Recalls, and Reflects](https://mlsanity.com/paper/2512.12818/) (2025)
- [A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents](https://mlsanity.com/paper/2602.06052/) (2026)
- [LongMemEval-V2: Evaluating Long-Term Agent Memory Toward Experienced Colleagues](https://mlsanity.com/paper/2605.12493/) (2026)
- [Evaluating Long-Horizon Memory for Multi-Party Collaborative Dialogues](https://mlsanity.com/paper/2602.01313/) (2026)
- [From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanisms](https://mlsanity.com/paper/2605.06716/) (2026)
- [PERMA: Benchmarking Personalized Memory Agents via Event-Driven Preference and Realistic Task Environments](https://mlsanity.com/paper/2603.23231/) (2026)
- [Towards Natural Personalization: Evaluating Long-Horizon Preference Following in Personalized User-LLM Interactions](https://mlsanity.com/paper/2603.04191/) (2026)
- [EMemBench: Interactive Benchmarking of Episodic Memory for VLM Agents](https://mlsanity.com/paper/2601.16690/) (2026)
- [SWE Context Bench: A Benchmark for Context Learning in Coding](https://mlsanity.com/paper/2602.08316/) (2026)
- [MemTxn: A Transaction Boundary for Source-Supported Updates and Complete-State Recovery in Agent Memory](https://mlsanity.com/paper/2607.27834/) (2026)
- [GoLongRL: Capability-Oriented Long Context Reinforcement Learning with Multitask Alignment](https://mlsanity.com/paper/2605.19577/) (2026)
- [MINTEval: Evaluating Memory under Multi-Target Interference in Long-Horizon Agent Systems](https://mlsanity.com/paper/2605.18565/) (2026)
- [Mitigating Provenance-Role Collapse in Long-Term Agents via Typed Memory Representation](https://mlsanity.com/paper/2605.25869/) (2026)
- [ElasticMem: Latent Memory as a Learnable Resource for LLM Agents](https://mlsanity.com/paper/2605.30690/) (2026)
- [MemTrace: Probing What Final Accuracy Misses in Long-Term Memory](https://mlsanity.com/paper/2606.17328/) (2026)
- [Context as an Environment: Programmatic Context Management for Long-Horizon Agents](https://mlsanity.com/paper/2608.21690/) (2026)
- [ATANT: An Evaluation Framework for AI Continuity](https://mlsanity.com/paper/2604.06710/) (2026)
- [Grounded Continuation: A Linear-Time Runtime Verifier for LLM Conversations](https://mlsanity.com/paper/2605.14175/) (2026)
- [MemDelta: Controlled Baselines and Hidden Confounds in Agent Memory Evaluation](https://mlsanity.com/paper/2606.29914/) (2026)
- [Mi-Memory: A Lifecycle Memory Framework for Personal AI](https://mlsanity.com/paper/2607.18975/) (2026)
- [Ground Truth First: A Longitudinal Evaluation Instrument for Agent Memory, and the Tenure Crossover in Memory-Architecture Rankings](https://mlsanity.com/paper/2607.21962/) (2026)
- [Fortunate Recall: Ontology-Driven Memory Lifecycle Management for Persistent Coherence in LLMs](https://mlsanity.com/paper/2609.10413/) (2026)
- [MemoryRewardBench: Benchmarking Reward Models for Long-Term Memory Management in Large Language Models](https://mlsanity.com/paper/2601.11969/) (2026)
- [Beyond Static Dialogues: Benchmarking Realistic, Heterogeneous, and Evolving Long-Term Memory](https://mlsanity.com/paper/2605.31086/) (2026)
- [SelfMem: Self-Optimizing Memory for AI Agents](https://mlsanity.com/paper/2607.03726/) (2026)
- [Oracle Agent Memory as an Enterprise Memory Substrate for Long-Horizon AI Agents](https://mlsanity.com/paper/2607.13157/) (2026)
- [Stop When Memory Suffices: Evidence-Conditioned Progressive Execution for LLM Agents](https://mlsanity.com/paper/2608.01285/) (2026)
- [Mnemon: Raw Records, Fast Judgments, Slow Thoughts](https://mlsanity.com/paper/2609.36059/) (2026)
- [ATANT v1.1: Positioning Continuity Evaluation Against Memory, Long-Context, and Agentic-Memory Benchmarks](https://mlsanity.com/paper/2604.10981/) (2026)
- [The Missing Knowledge Layer in Cognitive Architectures for AI Agents](https://mlsanity.com/paper/2604.11364/) (2026)

## Abstract (arXiv)

Evaluating the abilities of large language models (LLMs) for tasks that require long-term memory and thus long-context reasoning, for example in conversational settings, is hampered by the existing benchmarks, which often lack narrative coherence, cover narrow domains, and only test simple recall-oriented tasks. This paper introduces a comprehensive solution to these challenges. First, we present a novel framework for automatically generating long (up to 10M tokens), coherent, and topically diverse conversations, accompanied by probing questions targeting a wide range of memory abilities. From this, we construct BEAM, a new benchmark comprising 100 conversations and 2,000 validated questions. Second, to enhance model performance, we propose LIGHT-a framework inspired by human cognition that equips LLMs with three complementary memory systems: a long-term episodic memory, a short-term working memory, and a scratchpad for accumulating salient facts. Our experiments on BEAM reveal that even LLMs with 1M token context windows (with and without retrieval-augmentation) struggle as dialogues lengthen. In contrast, LIGHT consistently improves performance across various models, achieving an average improvement of 3.5%-12.69% over the strongest baselines, depending on the backbone LLM. An ablation study further confirms the contribution of each memory component.