mlsanityData to 14 Oct 2026

AgentMemBench: A Systematic Benchmark for Evaluating Long-Term Memory Management Strategies in Conversational AI Agents

arXiv 2608.00009 ↗First posted 16 Jun 2026cs.CL cs.AIcited by 0

On the map: LLM agents and tool use → Agent memory systems

Abstract (arXiv)

Long-term memory remains a critical bottleneck for conversational AI agents, whose finite context windows cannot support coherent recall across thousands of turns. We present AgentMemBench, a unified, reproducible benchmark evaluating five memory management strategies under identical conditions: in-context windowing (ICW), external key-value store (EKV), graph-based episodic memory (GEM), compression-based summarisation (CBS), and web-augmented memory (WAM). All are assessed across three public datasets covering long-term multi-session dialogue (LoCoMo), task-oriented document grounding (MultiDoc2Dial), and persona-grounded multi-session chat (MSC), using Recall@k, MRR, nDCG@k, Answer F1, an LLM-judge Faithfulness score, Memory Footprint, and Latency over 491 annotated question turns. Generation and judging both use Qwen2.5-7B-Instruct (4-bit), with greedy decoding for determinism. Our results show that (1) EKV dominates on every quality axis (macro Recall@5 0.792, MRR 0.677, F1 0.156, Faithfulness 0.354); (2) long-range recall is decisive: on LoCoMo, where the gold turn lies many sessions back, ICW, WAM, GEM, and CBS retrieve almost nothing (Recall@5 <= 0.005) while EKV alone reaches 0.573, showing that recency windows, summaries, and entity graphs collapse at long horizons and only dense retrieval scales; (3) CBS is the runner-up on retrieval (0.556); (4) WAM equals ICW on in-corpus recall by construction, since external results carry no in-corpus provenance; and (5) EKV's recall advantage carries a footprint cost (~5,100 vs ~300 tokens for ICW/WAM), an explicit accuracy-efficiency trade-off. We additionally evaluate two published memory systems (MemGPT/Letta, HippoRAG) against the same harness, and release all code, environment, and result artefacts for full reproducibility.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Agent memory systems, one of 16 directions in LLM agents and tool use. Builds on 25 of its 49 references that are in the map. First posted on arXiv 16 Jun 2026.

Builds on

25 of 49 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions7 Jul
  2. 2025In Prospect and Retrospect: Reflective Memory Management for Long-term Personalized Dialogue Agents11 Mar
  3. 2025A-MEM: Agentic Memory for LLM Agents17 Feb
  4. 2024Evaluating Very Long-Term Conversational Memory of LLM Agents27 Feb

Other references · 21

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  3. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  4. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  5. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarizationcited by 1,453
  6. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  7. 2023MemGPT: Towards LLMs as Operating Systemscited by 1,389
  8. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  9. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 707
  10. 2021Recursively Summarizing Books with Human Feedbackcited by 290
  11. 2024HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Modelscited by 281
  12. 2021Beyond Goldfish Memory: Long-Term Open-Domain Conversationcited by 267
9 more
  1. 2025From RAG to Memory: Non-Parametric Continual Learning for Large Language Modelscited by 231
  2. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 170
  3. 2023ChatDB: Augmenting LLMs with Databases as Their Symbolic Memorycited by 140
  4. 2025MemOS: A Memory OS for AI Systemcited by 138
  5. 2025From Human Memory to AI Memory: A Survey on Memory Mechanisms in the Era of LLMscited by 124
  6. 2024RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systemscited by 117
  7. 2023Prompted LLMs as Chatbot Modules for Long Open-domain Conversationcited by 61
  8. 2025Beyond a Million Tokens: Benchmarking and Enhancing Long-Term Memory in LLMscited by 60
  9. 2021MultiDoc2Dial: Modeling Dialogues Grounded in Multiple Documentscited by 53

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026MemOps: Benchmarking Lifecycle Memory Operations in Long-Horizon Conversations
  2. 2025Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory
  3. 2026APEX-MEM: Agentic Semi-Structured Memory with Temporal Reasoning for Long-Term Conversational AI
  4. 2026When Does Memory Help? A Cost-Aware Evaluation of Long-Term Memory in Tool-Using LLM Agents
  5. 2026AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications
  6. 2026Memory in the LLM Era: Modular Architectures and Strategies in a Unified Framework
  7. 2026From Recall to Forgetting: Benchmarking Long-Term Memory for Personalized Agents
  8. 2025SGMem: Sentence Graph Memory for Long-Term Conversational Agents