mlsanityData to 14 Oct 2026

EvalMem: An Operation-Level Diagnostic Framework for Long-Term Memory Systems

arXiv 2609.22231 ↗First posted 3 Sep 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Long-horizon interactions with LLM-based assistants require memory systems that preserve and update user states, preferences, and interaction histories. Existing evaluations report end-to-end QA accuracy and cannot determine whether errors arise from encoding, retrieval, or generation. We introduce EvalMem, an operation-level diagnostic framework with three parallel Examiners. For each query, the Encoding Examiner checks whether the target fact is stored, the Retrieval Examiner assesses whether the native retriever returns usable evidence, and the Generation Examiner tests whether the model can answer from oracle evidence. Their outputs form fine-grained multi-label defect codes. To improve store-level diagnosis, we adapt agentic RAG with a recall-first strategy that searches using both the query and source evidence, increasing recall of present evidence on LoCoMo from 70.2% to 95.6%. Evaluations of seven memory systems on LoCoMo, LongMemEval-S, and dynamic DynaMem-Bench identify retrieval as the most frequently attributed failure layer; in default LoCoMo, retrieval defects reach 22.1%, compared with 7.7% for encoding and 6.5% for generation. Guided by this diagnosis, MemWiki, a search-friendly auxiliary structure built from each system's memory export, improves mean accuracy by 2.5 and 2.3 percentage points on LoCoMo and LongMemEval-S, respectively.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 26 of its 53 references that are in the map. First posted on arXiv 3 Sep 2026.

Builds on

26 of 53 references.

References · 26

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  3. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  4. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  5. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  6. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  7. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
  8. 2025A-MEM: Agentic Memory for LLM Agentscited by 1,063
  9. 2022Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questionscited by 923
  10. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 707
  11. 2025Zep: A Temporal Knowledge Graph Architecture for Agent Memorycited by 400
  12. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 316
14 more
  1. 2025Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactionscited by 254
  2. 2024BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystackcited by 215
  3. 2024Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreementcited by 105
  4. 2025MemoryBench: A Benchmark for Memory and Continual Learning in LLM Systemscited by 61
  5. 2025HaluMem: Evaluating Hallucinations in Memory Systems of Agentscited by 54
  6. 2025General Agentic Memory Via Deep Researchcited by 47
  7. 2024Ada-LEval: Evaluating long-context LLMs with length-adaptable benchmarkscited by 32
  8. 2025O-Mem: Omni Memory System for Personalized, Long Horizon, Self-Evolving Agentscited by 32
  9. 2026TiMem: Temporal-Hierarchical Memory Consolidation for Long-Horizon Conversational Agentscited by 31
  10. 2026AMemGym: Interactive Memory Benchmarking for Assistants in Long-Horizon Conversationscited by 20
  11. 2026EvoMemBench: Benchmarking Agent Memory from a Self-Evolving Perspectivecited by 17
  12. 2024XL2Bench: A Benchmark for Extremely Long Context Understanding with Long-range Dependenciescited by 14
  13. 2026Membox: Weaving Topic Continuity into Long-Range Memory for LLM Agentscited by 6
  14. 2026MemGround: Long-Term Memory Evaluation Kit for Large Language Models in Gamified Scenarioscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Diagnosing Retrieval vs. Utilization Bottlenecks in LLM Agent Memory
  2. 2026PACMI: Provenance-Aware Cascading Memory Invalidation for Long-Term LLM Agents
  3. 2026D2ACCI: A Dual-Loop Diagnostic Protocol for Evidence-Preserving Agent Memory
  4. 2026VitaBench: Memory Lifecycles Beyond Retrieval
  5. 2026MemLens: A Value-Aware Memory Management System with Interactive Analytics for LLM-based Agents
  6. 2026A-TMA: Decoupling State-Aware Memory Failures in Long-Term Agent Memory
  7. 2026When Correct Memory Goes Wrong: Fuzzing Persistent Memory Use in LLM Agents
  8. 2026MEMAUDIT: An Exact Package-Oracle Evaluation Protocol for Budgeted Long-Term LLM Memory Writing