mlsanityData to 14 Oct 2026

Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions

arXiv 2507.05257 ↗First posted 7 Jul 2025cs.CL cs.AIcited by 254

On the map: LLM agents and tool use → Agent memory systems

Abstract (arXiv)

Recent benchmarks for Large Language Model (LLM) agents primarily focus on evaluating reasoning, planning, and execution capabilities, while another critical component-memory, encompassing how agents memorize, update, and retrieve long-term information-is under-evaluated due to the lack of benchmarks. We term agents with memory mechanisms as memory agents. In this paper, based on classic theories from memory science and cognitive science, we identify four core competencies essential for memory agents: accurate retrieval, test-time learning, long-range understanding, and selective forgetting. Existing benchmarks either rely on limited context lengths or are tailored for static, long-context settings like book-based QA, which do not reflect the interactive, multi-turn nature of memory agents that incrementally accumulate information. Moreover, no existing benchmarks cover all four competencies. We introduce MemoryAgentBench, a new benchmark specifically designed for memory agents. Our benchmark transforms existing long-context datasets and incorporates newly constructed datasets into a multi-turn format, effectively simulating the incremental information processing characteristic of memory agents. By carefully selecting and curating datasets, our benchmark provides comprehensive coverage of the four core memory competencies outlined above, thereby offering a systematic and challenging testbed for assessing memory quality. We evaluate a diverse set of memory agents, ranging from simple context-based and retrieval-augmented generation (RAG) systems to advanced agents with external memory modules and tool integration. Empirical results reveal that current methods fall short of mastering all four competencies, underscoring the need for further research into comprehensive memory mechanisms for LLM agents.

Key facts. Cited by 254 papers in the map (data through 14 Oct 2026). Placed in Agent memory systems, one of 16 directions in LLM agents and tool use. Builds on 69 of its 94 references that are in the map. First posted on arXiv 7 Jul 2025.

Builds on

69 of 94 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025MEM1: Learning to Synergize Memory and Reasoning for Efficient Long-Horizon Agents18 Jun
  2. 2025Memory OS of AI Agent30 May
  3. 2025Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory28 Apr
  4. 2025A-MEM: Agentic Memory for LLM Agents17 Feb
  5. 2025Zep: A Temporal Knowledge Graph Architecture for Agent Memory20 Jan
  6. 2024Evaluating Very Long-Term Conversational Memory of LLM Agents27 Feb

Other references · 63

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2024GPT-4o System Cardcited by 4,588
  3. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  5. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  6. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,453
  7. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  8. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
  9. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  10. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,261
  11. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  12. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,216
38 more
  1. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  2. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 1,002
  3. 2022Mass-Editing Memory in a Transformercited by 1,001
  4. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 707
  5. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 573
  6. 2023MemoryBank: Enhancing Large Language Models with Long-Term Memorycited by 556
  7. 2024RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrievalcited by 532
  8. 2024NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Modelscited by 472
  9. 2022Memory-Based Model Editing at Scalecited by 447
  10. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 433
  11. 2019An Evaluation Dataset for Intent Classification and Out-of-Scope Predictioncited by 386
  12. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  13. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  14. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 318
  15. 2023MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questionscited by 300
  16. 2023RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Modelscited by 283
  17. 2025MemAgent: Reshaping Long-Context LLM with Multi-Conv RL-based Memory Agentcited by 246
  18. 2024AlphaEdit: Null-Space Constrained Knowledge Editing for Language Modelscited by 236
  19. 2025From RAG to Memory: Non-Parametric Continual Learning for Large Language Modelscited by 231
  20. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 202
  21. 2018Towards Deep Conversational Recommendationscited by 176
  22. 2019Benchmarking Natural Language Understanding Services for building Conversational Agentscited by 166
  23. 2024Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generationcited by 162
  24. 2025OWL: Optimized Workforce Learning for General Multi-Agent Assistance in Real-World Task Automationcited by 131
  25. 2024In-Context Learning with Long-Context Models: An In-Depth Explorationcited by 128
  26. 2023Large Language Models as Zero-Shot Conversational Recommenderscited by 127
  27. 2025NoLiMa: Long-Context Evaluation Beyond Literal Matchingcited by 126
  28. 2024HELMET: How to Evaluate Long-Context Language Models Effectively and Thoroughlycited by 125
  29. 2024Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QAcited by 122
  30. 2024MemoRAG: Boosting Long Context Processing with Global Memory-Enhanced Retrieval Augmentationcited by 119
  31. 2024RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systemscited by 117
  32. 2024MEMORYLLM: Towards Self-Updatable Large Language Modelscited by 93
  33. 2023Weakly-Supervised Concealed Object Segmentation with SAM-based Pseudo Labeling and Multi-scale Feature Groupingcited by 92
  34. 2024One Thousand and One Pairs: A "novel" challenge for long-context language modelscited by 80
  35. 2024Diffusion Models in Low-Level Vision: A Surveycited by 74
  36. 2025M+: Extending MemoryLLM with Scalable Long-Term Memorycited by 54
  37. 2024CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Modelscited by 50
  38. 2024Auto-RAG: Autonomous Retrieval-Augmented Generation for Large Language Modelscited by 43

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 19; 2026: 235.

254 papers.

Sort:

By area:

  1. 2025Memory in the Age of AI Agentscited by 303
  2. 2025ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memorycited by 284
  3. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  4. 2025LightMem: Lightweight and Efficient Memory-Augmented Generationcited by 180
  5. 2025Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memorycited by 136
  6. 2026AutoSkill: Experience-Driven Lifelong Learning via Skill Self-Evolutioncited by 115
  7. 2026MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Taskscited by 84
  8. 2026Agentic Memory: Learning Unified Long-Term and Short-Term Memory Management for Large Language Model Agentscited by 78
  9. 2026Memory for Autonomous LLM Agents:Mechanisms, Evaluation, and Emerging Frontierscited by 65
  10. 2025MemoryBench: A Benchmark for Memory and Continual Learning in LLM Systemscited by 61
  11. 2025Mem-α: Learning Memory Construction via Reinforcement Learningcited by 55
  12. 2026Mem-Gallery: Benchmarking Multimodal Long-Term Conversational Memory for MLLM Agentscited by 54
38 more
  1. 2026AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applicationscited by 53
  2. 2025LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directionscited by 50
  3. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 50
  4. 2026STALE: Can LLM Agents Know When Their Memories Are No Longer Valid?cited by 50
  5. 2026A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agentscited by 44
  6. 2026From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agentscited by 36
  7. 2026Mem2ActBench: A Benchmark for Evaluating Long-Term Memory Utilization in Task-Oriented Autonomous Agentscited by 32
  8. 2026A Survey on Long-Term Memory Security in LLM Agents: Attacks, Defenses, and Governance Across the Memory Lifecyclecited by 27
  9. 2026Graph-based Agent Memory: Taxonomy, Techniques, and Applicationscited by 25
  10. 2025IterResearch: Rethinking Long-Horizon Agents with Interaction Scalingcited by 23
  11. 2026REMem: Reasoning with Episodic Memory in Language Agentcited by 23
  12. 2026OP-Bench: Benchmarking Over-Personalization for Memory-Augmented Personalized Conversational Agentscited by 22
  13. 2026From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanismscited by 21
  14. 2026AMemGym: Interactive Memory Benchmarking for Assistants in Long-Horizon Conversationscited by 20
  15. 2026Supersede: Diagnosing and Training the Memory-Update Gap in LLM Agentscited by 20
  16. 2026From Lossy to Verified: A Provenance-Aware Tiered Memory for Agentscited by 17
  17. 2026MemoryCD: Benchmarking Long-Context User Memory of LLM Agents for Lifelong Cross-Domain Personalizationcited by 17
  18. 2026GroupMemBench: Benchmarking LLM Agent Memory in Multi-Party Conversationscited by 17
  19. 2026EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspectivecited by 17
  20. 2026MemMark: State-Evolution Attribution Watermarking for Agent Long-Term Memory Systemscited by 16
  21. 2026Are We Ready For An Agent-Native Memory System?cited by 15
  22. 2026Toward Personalized LLM-Powered Agents: Foundations, Evaluation, and Future Directionscited by 14
  23. 2026The Evolution of Tool Use in LLM Agents: From Single-Tool Call to Multi-Tool Orchestrationcited by 14
  24. 2026Jev-Mem: System-One-Controlled Agentic Memory for Efficient AI Agentscited by 14
  25. 2025MEMTRACK: Evaluating Long-Term Memory and State Tracking in Multi-Platform Dynamic Agent Environmentscited by 13
  26. 2026EgoMemReason: A Memory-Driven Reasoning Benchmark for Long-Horizon Egocentric Video Understandingcited by 13
  27. 2026δ-mem: Efficient Online Memory for Large Language Modelscited by 13
  28. 2026GateMem: Benchmarking Memory Governance in Multi-Principal Shared-Memory Agentscited by 13
  29. 2025AI Meets Brain: Memory Systems from Cognitive Neuroscience to Autonomous Agentscited by 12
  30. 2026Evaluating Memory Structure in LLM Agentscited by 12
  31. 2026Memory in the LLM Era: Modular Architectures and Strategies in a Unified Frameworkcited by 12
  32. 2026Trojan Hippo: Weaponizing Agent Memory for Data Exfiltrationcited by 12
  33. 2026Beyond Similarity: Trustworthy Memory Search for Personal AI Agentscited by 12
  34. 2025AssoMem: Scalable Memory QA with Multi-Signal Associative Retrievalcited by 11
  35. 2026KnowMe-Bench: Benchmarking Person Understanding for Lifelong Digital Companionscited by 11
  36. 2026Belief Memory: Agent Memory Under Partial Observabilitycited by 11
  37. 2025Convomem Benchmark: Why Your First 150 Conversations Don't Need RAGcited by 10
  38. 2026The Memory Curse: How Expanded Recall Erodes Cooperative Intent in LLM Agentscited by 10

50 most cited of 254 All 254 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Rethinking Memory in LLM based Agents: Representations, Operations, and Emerging Topics
  2. 2024On the Structural Memory of LLM Agents
  3. 2025Intrinsic Memory Agents: Heterogeneous Multi-Agent LLM Systems through Structured Contextual Memory
  4. 2026AdMem: Advanced Memory for Task-solving Agents
  5. 2024A Survey on the Memory Mechanism of Large Language Model based Agents
  6. 2026Weighted Memory Tree: Remembering What Matters for Long-Horizon LLM Agents
  7. 2026Anatomy of Agentic Memory: Taxonomy and Empirical Analysis of Evaluation and System Limitations
  8. 2025Hierarchical Memory for High-Efficiency Long-Term Reasoning in LLM Agents