# Episodic Memories Generation and Evaluation Benchmark for Large Language Models

arXiv: https://arxiv.org/abs/2501.13121 · HTML: https://mlsanity.com/paper/2501.13121/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 91 references that are in the map.
- First posted on arXiv 21 Jan 2025.

## Builds on (most cited first, 30 of 40)

- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Generating Long Sequences with Sparse Transformers](https://mlsanity.com/paper/1904.10509/) (2019)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [RULER: What's the Real Context Size of Your Long-Context Language Models?](https://mlsanity.com/paper/2404.06654/) (2024)
- [Leveraging Passage Retrieval with Generative Models for Open Domain Question Answering](https://mlsanity.com/paper/2007.01282/) (2020)
- [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022)
- [The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks](https://mlsanity.com/paper/1802.08232/) (2018)
- [Generalization through Memorization: Nearest Neighbor Language Models](https://mlsanity.com/paper/1911.00172/) (2019)
- [Quantifying Memorization Across Neural Language Models](https://mlsanity.com/paper/2202.07646/) (2022)
- [Knowledge Neurons in Pretrained Transformers](https://mlsanity.com/paper/2104.08696/) (2021)
- [KILT: a Benchmark for Knowledge Intensive Language Tasks](https://mlsanity.com/paper/2009.02252/) (2020)
- [Large Language Models Struggle to Learn Long-Tail Knowledge](https://mlsanity.com/paper/2211.08411/) (2022)
- [∞Bench: Extending Long Context Evaluation Beyond 100K Tokens](https://mlsanity.com/paper/2402.13718/) (2024)
- [Towards AI-Complete Question Answering: A Set of Prerequisite Toy Tasks](https://mlsanity.com/paper/1502.05698/) (2015)
- [Time-Aware Language Models as Temporal Knowledge Bases](https://mlsanity.com/paper/2106.15110/) (2021)
- [RealTime QA: What's the Answer Right Now?](https://mlsanity.com/paper/2207.13332/) (2022)
- [Modifying Memories in Transformer Models](https://mlsanity.com/paper/2012.00363/) (2020)
- [BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack](https://mlsanity.com/paper/2406.10149/) (2024)
- [Many-Shot In-Context Learning](https://mlsanity.com/paper/2404.11018/) (2024)
- [LongNet: Scaling Transformers to 1,000,000,000 Tokens](https://mlsanity.com/paper/2307.02486/) (2023)
- [Augmenting Language Models with Long-Term Memory](https://mlsanity.com/paper/2306.07174/) (2023)
- [Sources of Hallucination by Large Language Models on Inference Tasks](https://mlsanity.com/paper/2305.14552/) (2023)
- [A Dataset for Answering Time-Sensitive Questions](https://mlsanity.com/paper/2108.06314/) (2021)
- [Emergent and Predictable Memorization in Large Language Models](https://mlsanity.com/paper/2304.11158/) (2023)
- [Should we really use post-hoc tests based on mean-ranks?](https://mlsanity.com/paper/1505.02288/) (2015)

## Built on by (most cited first, 14 of 14)

- [(How) Do Language Models Track State?](https://mlsanity.com/paper/2503.02854/) (2025)
- [Embodied Agents Meet Personalization: Investigating Challenges and Solutions Through the Lens of Memory Utilization](https://mlsanity.com/paper/2505.16348/) (2025)
- [Persona2Web: Benchmarking Personalized Web Agents for Contextual Reasoning with User History](https://mlsanity.com/paper/2602.17003/) (2026)
- [Evolutionary Perspectives on the Evaluation of LLM-Based AI Agents: A Comprehensive Survey](https://mlsanity.com/paper/2506.11102/) (2025)
- [Memory in Large Language Models: Mechanisms, Evaluation and Evolution](https://mlsanity.com/paper/2509.18868/) (2025)
- [LMEB: Long-horizon Memory Embedding Benchmark](https://mlsanity.com/paper/2603.12572/) (2026)
- [RaMem: Contextual Reinstatement for Long-term Agentic Memory](https://mlsanity.com/paper/2606.22844/) (2026)
- [Knowing the Answer Isn't Enough: Fixing Reasoning Path Failures in LVLMs](https://mlsanity.com/paper/2512.06258/) (2025)
- [The Behavioral Fabric of LLM-Powered GUI Agents: Human Values and Interaction Outcomes](https://mlsanity.com/paper/2601.16356/) (2026)
- [Fire360: A Benchmark for Robust Perception and Episodic Memory in Degraded 360-Degree Firefighting Videos](https://mlsanity.com/paper/2506.02167/) (2025)
- [Explainability Through Systematicity: The Hard Systematicity Challenge for Artificial Intelligence](https://mlsanity.com/paper/2507.22197/) (2025)
- [Word2Spike: Poisson Rate Coding for Associative Memories and Neuromorphic Algorithms](https://mlsanity.com/paper/2509.07361/) (2025)
- [Diagnostics of cognitive failures in multi-agent expert systems using dynamic evaluation protocols and subsequent mutation of the processing context](https://mlsanity.com/paper/2509.15366/) (2025)
- [Synthetic Worlds for Temporal Evaluation and Knowledge Updating in LLMs](https://mlsanity.com/paper/2609.00184/) (2026)

## Abstract (arXiv)

Episodic memory -- the ability to recall specific events grounded in time and space -- is a cornerstone of human cognition, enabling not only coherent storytelling, but also planning and decision-making. Despite their remarkable capabilities, Large Language Models (LLMs) lack a robust mechanism for episodic memory: we argue that integrating episodic memory capabilities into LLM is essential for advancing AI towards human-like cognition, increasing their potential to reason consistently and ground their output in real-world episodic events, hence avoiding confabulations. To address this challenge, we introduce a comprehensive framework to model and evaluate LLM episodic memory capabilities. Drawing inspiration from cognitive science, we develop a structured approach to represent episodic events, encapsulating temporal and spatial contexts, involved entities, and detailed descriptions. We synthesize a unique episodic memory benchmark, free from contamination, and release open source code and datasets to assess LLM performance across various recall and episodic reasoning tasks. Our evaluation of state-of-the-art models, including GPT-4 and Claude variants, Llama 3.1, and o1-mini, reveals that even the most advanced LLMs struggle with episodic memory tasks, particularly when dealing with multiple related events or complex spatio-temporal relationships -- even in contexts as short as 10k-100k tokens.