mlsanityData to 14 Oct 2026

MemoryBench: A Benchmark for Memory and Continual Learning in LLM Systems

arXiv 2510.17281 ↗First posted 20 Oct 2025cs.LG cs.AI cs.IRcited by 61

Not on the map. Why?

Cited by 35 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Scaling up data, parameters, and test-time computation has been the mainstream methods to improve LLM systems (LLMsys), but their upper bounds are almost reached due to the gradual depletion of high-quality data and marginal gains obtained from larger computational resource consumption. Inspired by the abilities of human and traditional AI systems in learning from practice, constructing memory and continual learning frameworks for LLMsys has become an important and popular research direction in recent literature. Yet, existing benchmarks for LLM memory often focus on evaluating the system on homogeneous reading comprehension tasks with long-form inputs rather than testing their abilities to learn from accumulated user feedback in service time. Therefore, we propose a user feedback simulation framework and a comprehensive benchmark covering multiple domains, languages, and types of tasks to evaluate the continual learning abilities of LLMsys. Experiments show that the effectiveness and efficiency of state-of-the-art baselines are far from satisfying, and we hope this benchmark could pave the way for future studies on LLM memory and optimization algorithms. Website: https://memorybench.thuir.cn Code: https://github.com/THUIR/MemoryBench Data: https://huggingface.co/datasets/THUIR/MemoryBench Data-Full: https://huggingface.co/datasets/THUIR/MemoryBench-Full

Key facts. Cited by 61 papers in the map (data through 14 Oct 2026). Builds on 42 of its 73 references that are in the map. First posted on arXiv 20 Oct 2025.

Builds on

42 of 73 references.

References · 42

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2025Qwen3 Technical Reportcited by 8,820
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  8. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  9. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
  10. 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,261
  11. 2018Hierarchical Neural Story Generationcited by 1,193
  12. 2025A-MEM: Agentic Memory for LLM Agentscited by 1,063
30 more
  1. 2024Evaluating Very Long-Term Conversational Memory of LLM Agentscited by 830
  2. 2025Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memorycited by 779
  3. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 707
  4. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 634
  5. 2023MemoryBank: Enhancing Large Language Models with Long-Term Memorycited by 556
  6. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 465
  7. 2025Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactionscited by 254
  8. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 234
  9. 2022BlenderBot 3: a deployed conversational agent that continually learns to responsibly engagecited by 198
  10. 2025Memory OS of AI Agentcited by 150
  11. 2025From Human Memory to AI Memory: A Survey on Memory Mechanisms in the Era of LLMscited by 124
  12. 2025WritingBench: A Comprehensive Benchmark for Generative Writingcited by 106
  13. 2023MemoChat: Tuning LLMs to Use Memos for Consistent Long-Range Open-Domain Conversationcited by 97
  14. 2025MemBench: Towards More Comprehensive Evaluation on the Memory of LLM-based Agentscited by 84
  15. 2023Think-in-Memory: Recalling and Post-thinking Enable LLMs with Long-Term Memorycited by 64
  16. 2024HelloBench: Evaluating Long Text Generation Capabilities of Large Language Modelscited by 44
  17. 2024LexEval: A Comprehensive Chinese Legal Benchmark for Evaluating Large Language Modelscited by 37
  18. 2024PerLTQA: A Personal Long-Term Memory Dataset for Memory Classification, Retrieval, and Synthesis in Question Answeringcited by 34
  19. 2023User Simulation for Evaluating Information Access Systemscited by 31
  20. 2025Cognitive Memory in Large Language Modelscited by 29
  21. 2023LARP: Language-Agent Role Play for Open-World Gamescited by 24
  22. 2024MemSim: A Bayesian Simulator for Evaluating Memory of LLM-based Personal Assistantscited by 24
  23. 2023An In-depth Investigation of User Response Simulation for Conversational Searchcited by 20
  24. 2025Can LLMs Identify Critical Limitations within Scientific Research? A Systematic Evaluation on AI Research Paperscited by 20
  25. 2025JuDGE: Benchmarking Judgment Document Generation for Chinese Legal Systemcited by 18
  26. 2025A Probabilistic Inference Scaling Theory for LLM Self-Correctioncited by 9
  27. 2025Chinese Court Simulation with LLM-Based Agent Systemcited by 8
  28. 2025JRE-L: Journalist, Reader, and Editor LLMs in the Loop for Science Journalism for the General Audiencecited by 3
  29. 2025Exploring Human-Like Thinking in Search Simulations with Large Language Modelscited by 3
  30. 2025Overview of the NTCIR-18 Automatic Evaluation of LLMs (AEOLLM) Taskcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 57.

61 papers.

Sort:

By area:

  1. 2025Memory in the Age of AI Agentscited by 303
  2. 2026MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Taskscited by 84
  3. 2026Meta Context Engineering via Agentic Skill Evolutioncited by 49
  4. 2025Hindsight is 20/20: Building Agent Memory that Retains, Recalls, and Reflectscited by 44
  5. 2026A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agentscited by 44
  6. 2026Skill Retrieval Augmentation for Agentic AIcited by 44
  7. 2026Graph-based Agent Memory: Taxonomy, Techniques, and Applicationscited by 25
  8. 2026From Storage to Experience: A Survey on the Evolution of LLM Agent Memory Mechanismscited by 21
  9. 2026Mem-T: Densifying Rewards for Long-Horizon Memory Agentscited by 18
  10. 2026Governing Evolving Memory in LLM Agents: Risks, Mechanisms, and the Stability and Safety Governed Memory (SSGM) Frameworkcited by 18
  11. 2026MemoryCD: Benchmarking Long-Context User Memory of LLM Agents for Lifelong Cross-Domain Personalizationcited by 17
  12. 2026GroupMemBench: Benchmarking LLM Agent Memory in Multi-Party Conversationscited by 17
38 more
  1. 2026EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspectivecited by 17
  2. 2026Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environmentscited by 17
  3. 2026PERMA: Benchmarking Personalized Memory Agents via Event-Driven Preference and Realistic Task Environmentscited by 14
  4. 2025AI Meets Brain: Memory Systems from Cognitive Neuroscience to Autonomous Agentscited by 12
  5. 2026Evaluating Memory Structure in LLM Agentscited by 12
  6. 2026MemSyco-Bench: Benchmarking Sycophancy in Agent Memorycited by 12
  7. 2026E-mem: Multi-agent based Episodic Context Reconstruction for LLM Agent Memorycited by 10
  8. 2026EMemBench: Interactive Benchmarking of Episodic Memory for VLM Agentscited by 9
  9. 2026AgentCL: Toward Rigorous Evaluation of Continual Learning in Language Agentscited by 9
  10. 2026SuperLocalMemory: Privacy-Preserving Multi-Agent Memory with Bayesian Trust Defense Against Memory Poisoningcited by 7
  11. 2026MemOps: Benchmarking Lifecycle Memory Operations in Long-Horizon Conversationscited by 7
  12. 2026SE-Bench: Benchmarking Self-Evolution with Knowledge Internalizationcited by 6
  13. 2026State Contamination in Memory-Augmented LLM Agentscited by 5
  14. 2026MemGym: a Long-Horizon Memory Environment for LLM Agentscited by 5
  15. 2025Beyond Heuristics: A Decision-Theoretic Framework for Agent Memory Managementcited by 4
  16. 2026MemTrace: Probing What Final Accuracy Misses in Long-Term Memorycited by 4
  17. 2026ATANT: An Evaluation Framework for AI Continuitycited by 3
  18. 2026MemFail: Stress-Testing Failure Modes of LLM Memory Systemscited by 3
  19. 2026MemDelta: Controlled Baselines and Hidden Confounds in Agent Memory Evaluationcited by 3
  20. 2026PATH-Bench: Path-Dependent Evaluation of Lifelong Agentscited by 3
  21. 2026MemoryRewardBench: Benchmarking Reward Models for Long-Term Memory Management in Large Language Modelscited by 2
  22. 2026Improve Large Language Model Systems with User Logscited by 1
  23. 2026ATANT v1.1: Positioning Continuity Evaluation Against Memory, Long-Context, and Agentic-Memory Benchmarkscited by 1
  24. 2026EXG: Self-Evolving Agents with Experience Graphscited by 1
  25. 2026Exploring Cross-Scenario Generality of Agentic Memory Systems: Diagnostics and a Strong Baselinecited by 1
  26. 2026LifeSide: Benchmarking Agents as Lifelong Digital Companionscited by 1
  27. 2026StreamMemBench: Streaming Evaluation of Agent Memory for Future-Oriented Assistancecited by 1
  28. 2026Beyond Perplexity: A Behavioral Evaluation Framework for Deployment-Memory Claims in LLM Test-Time Trainingcited by 1
  29. 2026When Classic Cache Policies Fail: Learning-Augmented Replacement for Semantic Retrieval Bufferscited by 1
  30. 2026KVShareArena: KV-Cache Reuse Across Contexts and Model Checkpointscited by 1
  31. 2026STEM Agent: A Self-Adapting, Tool-Enabled, Extensible Architecture for Multi-Protocol AI Agent Systemscited by 0
  32. 2026Trust Your Memory: Verifiable Control of Smart Homes through Reinforcement Learning with Multi-dimensional Rewardscited by 0
  33. 2026Toward Measuring Structural Drift in LLM Communication Loopscited by 0
  34. 2026Detecting Clinical Discrepancies in Health Coaching Agents: A Dual-Stream Memory and Reconciliation Architecturecited by 0
  35. 2026Connecting the Dots: Benchmarking Reflective Memory in Long-Horizon Dialoguecited by 0
  36. 2026Decoupled Mixture-of-Experts for Parametric Knowledge Injectioncited by 0
  37. 2026Beyond the Current Observation: Evaluating Multimodal Large Language Models in Controllable Non-Markov Gamescited by 0
  38. 2026MemArena: An Ego-Centric Benchmark for On-Device Agentic Personal Memory Assistants at Scalecited by 0

50 most cited of 61 All 61 (JSON)