mlsanityData to 7 Oct 2026

MemSim: A Bayesian Simulator for Evaluating Memory of LLM-based Personal Assistants

arXiv 2409.20163 ↗First posted 30 Sep 2024cs.AI cs.CLcited by 24

On the map: LLM agents and tool use → Agent memory systems

Abstract (arXiv)

LLM-based agents have been widely applied as personal assistants, capable of memorizing information from user messages and responding to personal queries. However, there still lacks an objective and automatic evaluation on their memory capability, largely due to the challenges in constructing reliable questions and answers (QAs) according to user messages. In this paper, we propose MemSim, a Bayesian simulator designed to automatically construct reliable QAs from generated user messages, simultaneously keeping their diversity and scalability. Specifically, we introduce the Bayesian Relation Network (BRNet) and a causal generation mechanism to mitigate the impact of LLM hallucinations on factual information, facilitating the automatic creation of an evaluation dataset. Based on MemSim, we generate a dataset in the daily-life scenario, named MemDaily, and conduct extensive experiments to assess the effectiveness of our approach. We also provide a benchmark for evaluating different memory mechanisms in LLM-based agents with the MemDaily dataset. To benefit the research community, we have released our project at https://github.com/nuster1128/MemSim.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in Agent memory systems, one of 16 directions in LLM agents and tool use. Builds on 28 of its 54 references that are in the map. First posted on arXiv 30 Sep 2024.

Builds on

28 of 54 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024A Survey on the Memory Mechanism of Large Language Model based Agents21 Apr
  2. 2024Evaluating Very Long-Term Conversational Memory of LLM Agents27 Feb

Other references · 26

  1. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  2. 2017Billion-scale similarity search with GPUscited by 2,477
  3. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  4. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  5. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  6. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,295
  7. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,096
  8. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 752
  9. 2023MemoryBank: Enhancing Large Language Models with Long-Term Memorycited by 539
  10. 2018The Web as a Knowledge-base for Answering Complex Questionscited by 409
  11. 2024Graph Retrieval-Augmented Generation: A Surveycited by 320
  12. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
14 more
  1. 2023ChatDB: Augmenting LLMs with Databases as Their Symbolic Memorycited by 136
  2. 2024Graph Chain-of-Thought: Augmenting Large Language Models by Reasoning on Graphscited by 116
  3. 2023Retroformer: Retrospective Large Language Agents with Policy Gradient Optimizationcited by 110
  4. 2023MemoChat: Tuning LLMs to Use Memos for Consistent Long-Range Open-Domain Conversationcited by 94
  5. 2024CRAG -- Comprehensive RAG Benchmarkcited by 94
  6. 2017Causal Structure Learningcited by 84
  7. 2021A Survey on Complex Knowledge Base Question Answering: Methods, Challenges and Solutionscited by 77
  8. 2020KQA Pro: A Dataset with Explicit Compositional Programs for Complex Question Answering over Knowledge Basecited by 76
  9. 2023RET-LLM: Towards a General Read-Write Memory for Large Language Modelscited by 64
  10. 2023Think-in-Memory: Recalling and Post-thinking Enable LLMs with Long-Term Memorycited by 64
  11. 2023Prompted LLMs as Chatbot Modules for Long Open-domain Conversationcited by 61
  12. 2023LLM as OS, Agents as Apps: Envisioning AIOS, Agents and the AIOS-Agent Ecosystemcited by 35
  13. 2024Prompting Large Language Models with Knowledge Graphs for Question Answering Involving Long-tail Factscited by 14
  14. 2024A Better LLM Evaluator for Text Generation: The Impact of Prompt Output Sequencing and Optimizationcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 12; 2026: 12.

24 papers.

Sort:

By area:

  1. 2025A Survey of Context Engineering for Large Language Modelscited by 120
  2. 2025MemBench: Towards More Comprehensive Evaluation on the Memory of LLM-based Agentscited by 82
  3. 2025A Survey of Personalized Large Language Models: Progress and Future Directionscited by 65
  4. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Surveycited by 56
  5. 2025MemoryBench: A Benchmark for Memory and Continual Learning in LLM Systemscited by 56
  6. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  7. 2026From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agentscited by 29
  8. 2025From Single to Multi-Granularity: Toward Long-Term Memory Association and Selection of Conversational Agentscited by 27
  9. 2025Learn to Memorize: Optimizing LLM-based Agents with Adaptive Memory Frameworkcited by 25
  10. 2026Graph-based Agent Memory: Taxonomy, Techniques, and Applicationscited by 22
  11. 2025Large language models for artificial general intelligence (AGI): A survey of foundational principles and approachescited by 18
  12. 2025Explicit v.s. Implicit Memory: Exploring Multi-hop Complex Reasoning Over Personalized Informationcited by 16
12 more
  1. 2025AI Meets Brain: Memory Systems from Cognitive Neuroscience to Autonomous Agentscited by 12
  2. 2025RecUserSim: A Realistic and Diverse User Simulator for Evaluating Conversational Recommender Systemscited by 11
  3. 2025Mitigating Memorization in LLMs using Activation Steeringcited by 7
  4. 2026How Does Personalized Memory Shape LLM Behavior? Benchmarking Rational Preference Utilization in Personalized Assistantscited by 7
  5. 2026Metis: Memory Foundation Modelcited by 5
  6. 2026MEMAUDIT: An Exact Package-Oracle Evaluation Protocol for Budgeted Long-Term LLM Memory Writingcited by 3
  7. 2026Anticipate and Learn: Unleashing Idle-Time Compute in Proactive Agentscited by 3
  8. 2026MemFail: Stress-Testing Failure Modes of LLM Memory Systemscited by 3
  9. 2026Opal: Private Memory for Personal AIcited by 1
  10. 2026Do Agents Dream of False Memories? Black-box Visual Attacks on Long-term Memory in Multimodal AI Agentscited by 1
  11. 2026DELTAMEM: Incremental Experience Memory for LLM Agents via Residual Treescited by 0
  12. 2026LifeFuse-Mem: Lifecycle-Aware State Fusion Against Temporary Overwriting for Long-Term Memorycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Evaluating Memory Structure in LLM Agents
  2. 2025MemInsight: Autonomous Memory Augmentation for LLM Agents
  3. 2026MemGym: a Long-Horizon Memory Environment for LLM Agents
  4. 2026AMemGym: Interactive Memory Benchmarking for Assistants in Long-Horizon Conversations
  5. 2026Towards Autonomous Memory Agents
  6. 2026Individual Turing Test: A Case Study of LLM-based Simulation Using Longitudinal Personal Data
  7. 2026MemArena: An Ego-Centric Benchmark for On-Device Agentic Personal Memory Assistants at Scale
  8. 2026Bi-Mem: Bidirectional Construction of Hierarchical Memory for Personalized LLMs via Inductive-Reflective Agents