mlsanityData to 7 Oct 2026

MemBench: Towards More Comprehensive Evaluation on the Memory of LLM-based Agents

arXiv 2506.21605 ↗First posted 20 Jun 2025cs.CL cs.AIcited by 82

On the map: LLM agents and tool use → Agent memory systems

Abstract (arXiv)

Recent works have highlighted the significance of memory mechanisms in LLM-based agents, which enable them to store observed information and adapt to dynamic environments. However, evaluating their memory capabilities still remains challenges. Previous evaluations are commonly limited by the diversity of memory levels and interactive scenarios. They also lack comprehensive metrics to reflect the memory capabilities from multiple aspects. To address these problems, in this paper, we construct a more comprehensive dataset and benchmark to evaluate the memory capability of LLM-based agents. Our dataset incorporates factual memory and reflective memory as different levels, and proposes participation and observation as various interactive scenarios. Based on our dataset, we present a benchmark, named MemBench, to evaluate the memory capability of LLM-based agents from multiple aspects, including their effectiveness, efficiency, and capacity. To benefit the research community, we release our dataset and project at https://github.com/import-myself/Membench.

Key facts. Cited by 82 papers in the map (data through 7 Oct 2026). Placed in Agent memory systems, one of 16 directions in LLM agents and tool use. Builds on 20 of its 26 references that are in the map. First posted on arXiv 20 Jun 2025.

Builds on

20 of 26 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025MemEngine: A Unified and Modular Library for Developing Advanced Memory of LLM-based Agents4 May
  2. 2024MemSim: A Bayesian Simulator for Evaluating Memory of LLM-based Personal Assistants30 Sep
  3. 2024A Survey on the Memory Mechanism of Large Language Model based Agents21 Apr
  4. 2024Evaluating Very Long-Term Conversational Memory of LLM Agents27 Feb

Other references · 16

  1. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  2. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  3. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  4. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  5. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
  6. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 656
  7. 2023A Survey on Large Language Models for Recommendationcited by 378
  8. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  9. 2023L-Eval: Instituting Standardized Evaluation for Long Context Language Modelscited by 232
  10. 2022Long Time No See! Open-Domain Conversation with Long-Term Persona Memorycited by 107
  11. 2023SCM: Enhancing Large Language Model with Self-Controlled Memory Frameworkcited by 98
  12. 2019Fine-Grained Spoiler Detection from Large-Scale Review Corporacited by 76
4 more
  1. 2024Large Language Model Agent in Financial Trading: A Surveycited by 60
  2. 2019Generating Personalized Recipes from Historical User Preferencescited by 52
  3. 2024PerLTQA: A Personal Long-Term Memory Dataset for Memory Classification, Retrieval, and Synthesis in Question Answeringcited by 34
  4. 2017Neural Personalized Response Generation as Domain Adaptationcited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 75.

82 papers.

Sort:

By area:

  1. 2026Memory for Autonomous LLM Agents:Mechanisms, Evaluation, and Emerging Frontierscited by 60
  2. 2025MemoryBench: A Benchmark for Memory and Continual Learning in LLM Systemscited by 56
  3. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  4. 2026A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agentscited by 40
  5. 2025GenEnv: Difficulty-Aligned Co-Evolution Between LLM Agents and Environment Simulatorscited by 34
  6. 2026From Untrusted Input to Trusted Memory: A Systematic Study of Memory Poisoning Attacks in LLM Agentscited by 23
  7. 2026Graph-based Agent Memory: Taxonomy, Techniques, and Applicationscited by 22
  8. 2025Explicit v.s. Implicit Memory: Exploring Multi-hop Complex Reasoning Over Personalized Informationcited by 16
  9. 2026MemoryCD: Benchmarking Long-Context User Memory of LLM Agents for Lifelong Cross-Domain Personalizationcited by 15
  10. 2026Mem-T: Densifying Rewards for Long-Horizon Memory Agentscited by 14
  11. 2026HorizonBench: Long-Horizon Personalization with Evolving Preferencescited by 14
  12. 2026Supersede: Diagnosing and Training the Memory-Update Gap in LLM Agentscited by 13
38 more
  1. 2026According to Me: Long-Term Personalized Referential Memory QAcited by 12
  2. 2026Lightweight LLM Agent Memory with Small Language Modelscited by 12
  3. 2026GroupMemBench: Benchmarking LLM Agent Memory in Multi-Party Conversationscited by 12
  4. 2026Toward Efficient Agents: Memory, Tool learning, and Planningcited by 11
  5. 2026EvolveMem:Self-Evolving Memory Architecture via AutoResearch for LLM Agentscited by 11
  6. 2026Are We Ready For An Agent-Native Memory System?cited by 11
  7. 2026The Memory Curse: How Expanded Recall Erodes Cooperative Intent in LLM Agentscited by 10
  8. 2026KnowMe-Bench: Benchmarking Person Understanding for Lifelong Digital Companionscited by 9
  9. 2026Memory-R2: Fair Credit Assignment for Long-Horizon Memory-Augmented LLM Agentscited by 9
  10. 2025Memory in Large Language Models: Mechanisms, Evaluation and Evolutioncited by 8
  11. 2026SWE Context Bench: A Benchmark for Context Learning in Codingcited by 8
  12. 2026GateMem: Benchmarking Memory Governance in Multi-Principal Shared-Memory Agentscited by 8
  13. 2025RGMem: Renormalization Group-inspired Memory Evolution for Language Agentscited by 7
  14. 2026EMemBench: Interactive Benchmarking of Episodic Memory for VLM Agentscited by 7
  15. 2026CloneMem: Benchmarking Long-Term Memory for AI Clonescited by 6
  16. 2026LMEB: Long-horizon Memory Embedding Benchmarkcited by 6
  17. 2026PM-Bench: Evaluating Prospective Memory in LLM Agentscited by 6
  18. 2026MemTrace: Tracing and Attributing Errors in Large Language Model Memory Systemscited by 5
  19. 2026MemTxn: A Transaction Boundary for Source-Supported Updates and Complete-State Recovery in Agent Memorycited by 5
  20. 2026Memory-Induced Tool-Drift in LLM Agentscited by 4
  21. 2026MemTrace: Probing What Final Accuracy Misses in Long-Term Memorycited by 4
  22. 2026Self-Evolving Agents as Dynamic Graph Transformation: A Survey and New Perspectivecited by 4
  23. 2026M2: Dual-Memory Augmentation for Long-Horizon Web Agents via Trajectory Summarization and Insight Retrievalcited by 3
  24. 2026ImplicitMemBench: Measuring Unconscious Behavioral Adaptation in Large Language Modelscited by 3
  25. 2026Evaluating Memory Capability in Continuous Lifelog Scenariocited by 3
  26. 2026Anticipate and Learn: Unleashing Idle-Time Compute in Proactive Agentscited by 3
  27. 2026MemFail: Stress-Testing Failure Modes of LLM Memory Systemscited by 3
  28. 2026Memory Retrieval for Changing Preferencescited by 3
  29. 2026H2HMem: A Multimodal Memory Benchmark for Agents in Human-Human Interactionscited by 3
  30. 2026DynamicMem: A Long-Horizon Memory Benchmark in Real-World Settingscited by 3
  31. 2026MemOps: Benchmarking Lifecycle Memory Operations in Long-Horizon Conversationscited by 3
  32. 2025Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agentscited by 2
  33. 2026MemoryRewardBench: Benchmarking Reward Models for Long-Term Memory Management in Large Language Modelscited by 2
  34. 2026Neuromem: A Granular Decomposition of the Streaming Lifecycle in External Memory for LLMscited by 2
  35. 2026Self-Evolving LLM Memory Extraction Across Heterogeneous Taskscited by 2
  36. 2026MEME: Multi-entity & Evolving Memory Evaluationcited by 2
  37. 2026Is Agent Memory a Database? Rethinking Data Foundations for Long-Term AI Agent Memorycited by 2
  38. 2026StateFuse: Deterministic Conflict-Preserving Memory for Multi-Agent Systemscited by 2

50 most cited of 82 All 82 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026MemGym: a Long-Horizon Memory Environment for LLM Agents
  2. 2026MemSyco-Bench: Benchmarking Sycophancy in Agent Memory
  3. 2026Memory in the LLM Era: Modular Architectures and Strategies in a Unified Framework
  4. 2026EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspective
  5. 2025Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions
  6. 2025How Memory Management Impacts LLM Agents: An Empirical Study of Experience-Following Behavior
  7. 2026Evaluating Memory Structure in LLM Agents
  8. 2026MemCalib: Benchmarking and Optimizing Memory Use in LLM Agents