mlsanityData to 14 Oct 2026

LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory

arXiv 2410.10813 ↗First posted 14 Oct 2024cs.CLcited by 707

Not on the map. Why?

Cited by 336 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent large language model (LLM)-driven chat assistant systems have integrated memory components to track user-assistant chat histories, enabling more accurate and personalized responses. However, their long-term memory capabilities in sustained interactions remain underexplored. We introduce LongMemEval, a comprehensive benchmark designed to evaluate five core long-term memory abilities of chat assistants: information extraction, multi-session reasoning, temporal reasoning, knowledge updates, and abstention. With 500 meticulously curated questions embedded within freely scalable user-assistant chat histories, LongMemEval presents a significant challenge to existing long-term memory systems, with commercial chat assistants and long-context LLMs showing a 30% accuracy drop on memorizing information across sustained interactions. We then present a unified framework that breaks down the long-term memory design into three stages: indexing, retrieval, and reading. Built upon key experimental insights, we propose several memory design optimizations including session decomposition for value granularity, fact-augmented key expansion for indexing, and time-aware query expansion for refining the search scope. Extensive experiments show that these optimizations greatly improve both memory recall and downstream question answering on LongMemEval. Overall, our study provides valuable resources and guidance for advancing the long-term memory capabilities of LLM-based chat assistants, paving the way toward more personalized and reliable conversational AI. Our benchmark and code are publicly available at https://github.com/xiaowu0162/LongMemEval.

Key facts. Cited by 707 papers in the map (data through 14 Oct 2026). Builds on 36 of its 50 references that are in the map. First posted on arXiv 14 Oct 2024.

Builds on

36 of 50 references.

References · 36

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  3. 2020Longformer: The Long-Document Transformercited by 3,413
  4. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  5. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  6. 2020Reformer: The Efficient Transformercited by 1,668
  7. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
  8. 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
  9. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  10. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 830
  11. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 792
  12. 2018MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modellingcited by 766
24 more
  1. 2018CoQA: A Conversational Question Answering Challengecited by 683
  2. 2022MTEB: Massive Text Embedding Benchmarkcited by 640
  3. 2023MemoryBank: Enhancing Large Language Models with Long-Term Memorycited by 556
  4. 2024RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrievalcited by 532
  5. 2023Learning to Compress Prompts with Gist Tokenscited by 390
  6. 2014Memory Networkscited by 363
  7. 2023Adapting Language Models to Compress Contextscited by 356
  8. 2023LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Modelscited by 287
  9. 2024HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Modelscited by 281
  10. 2021Beyond Goldfish Memory: Long-Term Open-Domain Conversationcited by 267
  11. 2022Memorizing Transformerscited by 226
  12. 2023Augmenting Language Models with Long-Term Memorycited by 189
  13. 2024Data Engineering for Scaling Language Models to 128K Contextcited by 186
  14. 2023Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Modelscited by 184
  15. 2021SituatedQA: Incorporating Extra-Linguistic Contexts into QAcited by 145
  16. 2023Walking Down the Memory Maze: Beyond Context Limit through Interactive Readingcited by 133
  17. 2024Hello Again! LLM-powered Personalized Agent for Long-term Dialoguecited by 123
  18. 2024Make Your LLM Fully Utilize the Contextcited by 119
  19. 2022Training Language Models with Memory Augmentationcited by 113
  20. 2022Long Time No See! Open-Domain Conversation with Long-Term Persona Memorycited by 108
  21. 2023Dense X Retrieval: What Retrieval Granularity Should We Use?cited by 88
  22. 2023Did You Read the Instructions? Rethinking the Effectiveness of Task Definitions in Instruction Learningcited by 41
  23. 2024PerLTQA: A Personal Long-Term Memory Dataset for Memory Classification, Retrieval, and Synthesis in Question Answeringcited by 34
  24. 2024Cognitive Kernel: An Open-source Agent System towards Generalist Autopilotscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 3; 2025: 81; 2026: 623.

707 papers.

Sort:

By area:

  1. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  2. 2025ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memorycited by 284
  3. 2025Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactionscited by 254
  4. 2025MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agentscited by 238
  5. 2025Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learningcited by 237
  6. 2025Survey on Evaluation of LLM-based Agentscited by 204
  7. 2025LightMem: Lightweight and Efficient Memory-Augmented Generationcited by 180
  8. 2025Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scalecited by 178
  9. 2025MIRIX: Multi-Agent Memory System for LLM-Based Agentscited by 173
  10. 2026SimpleMem: Efficient Lifelong Memory for LLM Agentscited by 149
  11. 2026MemSkill: Learning and Evolving Memory Skills for Self-Evolving Agentscited by 142
  12. 2025MemOS: A Memory OS for AI Systemcited by 138
38 more
  1. 2025Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memorycited by 136
  2. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  3. 2026AutoSkill: Experience-Driven Lifelong Learning via Skill Self-Evolutioncited by 115
  4. 2025In Prospect and Retrospect: Reflective Memory Management for Long-term Personalized Dialogue Agentscited by 106
  5. 2025PersonaMem-v2: Towards Personalized Intelligence via Learning Implicit User Personas and Agentic Memorycited by 88
  6. 2025MemBench: Towards More Comprehensive Evaluation on the Memory of LLM-based Agentscited by 84
  7. 2026MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Taskscited by 84
  8. 2025A Survey of Personalized Large Language Models: Progress and Future Directionscited by 71
  9. 2025MemoryBench: A Benchmark for Memory and Continual Learning in LLM Systemscited by 61
  10. 2025Mem-α: Learning Memory Construction via Reinforcement Learningcited by 55
  11. 2025HaluMem: Evaluating Hallucinations in Memory Systems of Agentscited by 54
  12. 2026EverMemOS: A Self-Organizing Memory Operating System for Structured Long-Horizon Reasoningcited by 54
  13. 2026Mem-Gallery: Benchmarking Multimodal Long-Term Conversational Memory for MLLM Agentscited by 54
  14. 2026AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applicationscited by 53
  15. 2026MAGMA: A Multi-Graph based Agentic Memory Architecture for AI Agentscited by 52
  16. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 50
  17. 2025What Deserves Memory: Adaptive Memory Distillation for LLM Agentscited by 48
  18. 2025Measuring What Makes You Unique: Difference-Aware User Modeling for Enhancing LLM Personalizationcited by 45
  19. 2025Hindsight is 20/20: Building Agent Memory that Retains, Recalls, and Reflectscited by 44
  20. 2025LEGOMem: Modular Procedural Memory for Multi-agent LLM Systems for Workflow Automationcited by 40
  21. 2025Oolong: Evaluating Long Context Reasoning and Aggregation Capabilitiescited by 40
  22. 2025LiCoMemory: Lightweight and Cognitive Agentic Memory for Efficient Long-Term Reasoningcited by 38
  23. 2025Rethinking Memory in LLM based Agents: Representations, Operations, and Emerging Topicscited by 36
  24. 2026From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agentscited by 36
  25. 2025MemVerse: Multimodal Memory for Lifelong Learning Agentscited by 35
  26. 2026From Untrusted Input to Trusted Memory: A Systematic Study of Memory Poisoning Attacks in LLM Agentscited by 33
  27. 2026Mem2ActBench: A Benchmark for Evaluating Long-Term Memory Utilization in Task-Oriented Autonomous Agentscited by 32
  28. 2026PlugMem: A Task-Agnostic Plugin Memory Module for LLM Agentscited by 32
  29. 2025Query-Focused Retrieval Heads Improve Long-Context Reasoning and Re-rankingcited by 31
  30. 2026TiMem: Temporal-Hierarchical Memory Consolidation for Long-Horizon Conversational Agentscited by 31
  31. 2025Think-While-Generating: On-the-Fly Reasoning for Personalized Long-Form Generationcited by 30
  32. 2025MMRC: A Large-Scale Benchmark for Understanding Multimodal Large Language Model in Real-World Conversationcited by 29
  33. 2025From Single to Multi-Granularity: Toward Long-Term Memory Association and Selection of Conversational Agentscited by 29
  34. 2025SGMem: Sentence Graph Memory for Long-Term Conversational Agentscited by 29
  35. 2025MemSearcher: Training LLMs to Reason, Search and Manage Memory via End-to-End Reinforcement Learningcited by 27
  36. 2026A Survey on Long-Term Memory Security in LLM Agents: Attacks, Defenses, and Governance Across the Memory Lifecyclecited by 27
  37. 2026Graph-based Agent Memory: Taxonomy, Techniques, and Applicationscited by 25
  38. 2025Symbiotic Cooperation for Web Agents: Harnessing Complementary Strengths of Large and Small LLMscited by 24

50 most cited of 707 All 707 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026RUMBA: Russian User Memory Benchmark
  2. 2026DyadMem: A Long-Term Memory Benchmark of How Agents Work with Users
  3. 2023Memory Sandbox: Transparent and Interactive Memory Management for Conversational Agents
  4. 2025Mem-PAL: Towards Memory-based Personalized Dialogue Assistants for Long-term User-Agent Interaction
  5. 2025Evaluating Long-Term Memory for Long-Context Question Answering
  6. 2025Mnemosyne: An Unsupervised, Human-Inspired Long-Term Memory Architecture for Edge-Based LLMs
  7. 2026OP-Bench: Benchmarking Over-Personalization for Memory-Augmented Personalized Conversational Agents
  8. 2026M2A: Multimodal Memory Agent with Dual-Layer Hybrid Memory for Long-Term Personalized Interactions