mlsanityData to 7 Oct 2026

Evaluating Very Long-Term Conversational Memory of LLM Agents

arXiv 2402.17753 ↗First posted 27 Feb 2024cs.CL cs.AI cs.LGcited by 800

On the map: LLM agents and tool use → Agent memory systems

Abstract (arXiv)

Existing works on long-term open-domain dialogues focus on evaluating model responses within contexts spanning no more than five chat sessions. Despite advancements in long-context large language models (LLMs) and retrieval augmented generation (RAG) techniques, their efficacy in very long-term dialogues remains unexplored. To address this research gap, we introduce a machine-human pipeline to generate high-quality, very long-term dialogues by leveraging LLM-based agent architectures and grounding their dialogues on personas and temporal event graphs. Moreover, we equip each agent with the capability of sharing and reacting to images. The generated conversations are verified and edited by human annotators for long-range consistency and grounding to the event graphs. Using this pipeline, we collect LoCoMo, a dataset of very long-term conversations, each encompassing 300 turns and 9K tokens on avg., over up to 35 sessions. Based on LoCoMo, we present a comprehensive evaluation benchmark to measure long-term memory in models, encompassing question answering, event summarization, and multi-modal dialogue generation tasks. Our experimental results indicate that LLMs exhibit challenges in understanding lengthy conversations and comprehending long-range temporal and causal dynamics within dialogues. Employing strategies like long-context LLMs or RAG can offer improvements but these models still substantially lag behind human performance.

Key facts. Cited by 800 papers in the map (data through 7 Oct 2026). Placed in Agent memory systems, one of 16 directions in LLM agents and tool use. Builds on 49 of its 71 references that are in the map. First posted on arXiv 27 Feb 2024.

Builds on

49 of 71 references.

References · 49

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  4. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  6. 2015VQA: Visual Question Answeringcited by 2,489
  7. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  8. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  9. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  10. 2023REPLUG: Retrieval-Augmented Black-Box Language Modelscited by 786
  11. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  12. 2021Retrieval Augmentation Reduces Hallucination in Conversationcited by 691
37 more
  1. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 649
  2. 2023Enabling Large Language Models to Generate Text with Citationscited by 584
  3. 2016Visual Dialogcited by 377
  4. 2023Evaluating Verifiability in Generative Search Enginescited by 297
  5. 2021Beyond Goldfish Memory: Long-Term Open-Domain Conversationcited by 266
  6. 2023LooGLE: Can Long-Context Language Models Understand Long Contexts?cited by 200
  7. 2021BookSum: A Collection of Datasets for Long-form Narrative Summarizationcited by 184
  8. 2018The Design and Implementation of XiaoIce, an Empathetic Social Chatbotcited by 172
  9. 2023AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotatorscited by 171
  10. 2022SODA: Million-scale Dialogue Distillation with Social Commonsense Contextualizationcited by 166
  11. 2020Open-Domain Question Answering Goes Conversational via Question Rewritingcited by 147
  12. 2023BooookScore: A systematic exploration of book-length summarization in the era of LLMscited by 137
  13. 2018Dialogue Natural Language Inferencecited by 135
  14. 2023Unlimiformer: Long-Range Transformers with Unlimited Length Inputcited by 127
  15. 2023How to Train Your DRAGON: Diverse Augmentation Towards Generalizable Dense Retrievalcited by 116
  16. 2023A Critical Evaluation of Evaluations for Long-form Question Answeringcited by 111
  17. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96
  18. 2023MemoChat: Tuning LLMs to Use Memos for Consistent Long-Range Open-Domain Conversationcited by 94
  19. 2018Image Chat: Engaging Grounded Conversationscited by 76
  20. 2020I like fish, especially dolphins: Addressing Contradictions in Dialogue Modelingcited by 68
  21. 2023Prompted LLMs as Chatbot Modules for Long Open-domain Conversationcited by 61
  22. 2017Image-Grounded Conversations: Multimodal Context for Natural Question and Response Generationcited by 59
  23. 2021DynaEval: Unifying Turn and Dialogue Level Evaluationcited by 59
  24. 2022MMDialog: A Large-scale Multi-turn Dialogue Dataset Towards Multi-modal Open-domain Conversationcited by 56
  25. 2023Conversation Chronicles: Towards Diverse Temporal and Relational Dynamics in Multi-Session Conversationscited by 47
  26. 2019CLEVR-Dialog: A Diagnostic Dataset for Multi-Round Reasoning in Visual Dialogcited by 43
  27. 2021PhotoChat: A Human-Human Dialogue Dataset with Photo Sharing Behavior for Joint Image-Text Modelingcited by 43
  28. 2021MMChat: Multi-Modal Chat Dataset on Social Mediacited by 30
  29. 2023Making Large Language Models Better Data Creatorscited by 28
  30. 2020OpenViDial: A Large-Scale, Open-Domain Dialogue Dataset with Visual Contextscited by 27
  31. 2022DEAM: Dialogue Coherence Evaluation using AMR-based Semantic Manipulationscited by 25
  32. 2021EmailSum: Abstractive Email Thread Summarizationcited by 24
  33. 2023MPCHAT: Towards Multimodal Persona-Grounded Conversationcited by 24
  34. 2023PeaCoK: Persona Commonsense Knowledge for Consistent and Engaging Narrativescited by 19
  35. 2022FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluationcited by 13
  36. 2023Mind the Gap Between Conversations for Improved Long-Term Dialogue Generationcited by 13
  37. 2019Sketch-Fill-A-R: A Persona-Grounded Chit-Chat Generation Frameworkcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 30; 2025: 130; 2026: 640.

800 papers.

Sort:

By area:

  1. 2025A-MEM: Agentic Memory for LLM Agentscited by 1,000
  2. 2025Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memorycited by 722
  3. 2024LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memorycited by 656
  4. 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 622
  5. 2025ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memorycited by 253
  6. 2025Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactionscited by 236
  7. 2025Memory-R1: Enhancing Large Language Model Agents to Manage and Utilize Memories via Reinforcement Learningcited by 220
  8. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  9. 2025Survey on Evaluation of LLM-based Agentscited by 190
  10. 2025Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scalecited by 163
  11. 2025LightMem: Lightweight and Efficient Memory-Augmented Generationcited by 163
  12. 2025MIRIX: Multi-Agent Memory System for LLM-Based Agentscited by 160
38 more
  1. 2025Evaluation and Benchmarking of LLM Agents: A Surveycited by 145
  2. 2025Memory OS of AI Agentcited by 140
  3. 2026SimpleMem: Efficient Lifelong Memory for LLM Agentscited by 140
  4. 2025MemOS: A Memory OS for AI Systemcited by 129
  5. 2025Evo-Memory: Benchmarking LLM Agent Test-time Learning with Self-Evolving Memorycited by 129
  6. 2026MemSkill: Learning and Evolving Memory Skills for Self-Evolving Agentscited by 129
  7. 2025ACON: Optimizing Context Compression for Long-horizon LLM Agentscited by 126
  8. 2024Hello Again! LLM-powered Personalized Agent for Long-term Dialoguecited by 121
  9. 2025NoLiMa: Long-Context Evaluation Beyond Literal Matchingcited by 120
  10. 2024Retrieval Augmented Generation or Long-Context LLMs? A Comprehensive Study and Hybrid Approachcited by 117
  11. 2026AutoSkill: Experience-Driven Lifelong Learning via Skill Self-Evolutioncited by 105
  12. 2025In Prospect and Retrospect: Reflective Memory Management for Long-term Personalized Dialogue Agentscited by 102
  13. 2024HiAgent: Hierarchical Working Memory Management for Solving Long-Horizon Agent Tasks with Large Language Modelcited by 101
  14. 2025On Memory Construction and Retrieval for Personalized Conversational Agentscited by 101
  15. 2024Flooding Spread of Manipulated Knowledge in LLM-Based Multi-Agent Communitiescited by 84
  16. 2025MemInsight: Autonomous Memory Augmentation for LLM Agentscited by 84
  17. 2025MemBench: Towards More Comprehensive Evaluation on the Memory of LLM-based Agentscited by 82
  18. 2025PersonaMem-v2: Towards Personalized Intelligence via Learning Implicit User Personas and Agentic Memorycited by 82
  19. 2026MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Taskscited by 72
  20. 2024What is Wrong with Perplexity for Long-context Language Modeling?cited by 68
  21. 2025A Survey of Personalized Large Language Models: Progress and Future Directionscited by 65
  22. 2025Hierarchical Memory for High-Efficiency Long-Term Reasoning in LLM Agentscited by 60
  23. 2026Memory for Autonomous LLM Agents:Mechanisms, Evaluation, and Emerging Frontierscited by 60
  24. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Surveycited by 56
  25. 2025MemoryBench: A Benchmark for Memory and Continual Learning in LLM Systemscited by 56
  26. 2025A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasurescited by 54
  27. 2026EverMemOS: A Self-Organizing Memory Operating System for Structured Long-Horizon Reasoningcited by 52
  28. 2025RAG vs. GraphRAG: A Systematic Evaluation and Key Insightscited by 51
  29. 2025HaluMem: Evaluating Hallucinations in Memory Systems of Agentscited by 51
  30. 2024Towards Lifelong Dialogue Agents via Timeline-based Memory Managementcited by 50
  31. 2025LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directionscited by 49
  32. 2025Mem-α: Learning Memory Construction via Reinforcement Learningcited by 49
  33. 2026MAGMA: A Multi-Graph based Agentic Memory Architecture for AI Agentscited by 49
  34. 2026Mem-Gallery: Benchmarking Multimodal Long-Term Conversational Memory for MLLM Agentscited by 49
  35. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  36. 2025What Deserves Memory: Adaptive Memory Distillation for LLM Agentscited by 46
  37. 2026AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applicationscited by 46
  38. 2025General Agentic Memory Via Deep Researchcited by 42

50 most cited of 800 All 800 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026AgenticAI-DialogGen: Topic-Guided Conversation Generation for Fine-Tuning and Evaluating Short- and Long-Term Memories of LLMs
  2. 2026EvolMem: A Cognitive-Driven Benchmark for Multi-Session Dialogue Memory
  3. 2024Should We Fine-Tune or RAG? Evaluating Different Techniques to Adapt LLMs for Dialogue
  4. 2026AdaMem: Adaptive User-Centric Memory for Long-Horizon Dialogue Agents
  5. 2024On the Benchmarking of LLMs for Open-Domain Dialogue Evaluation
  6. 2026APEX-MEM: Agentic Semi-Structured Memory with Temporal Reasoning for Long-Term Conversational AI
  7. 2026IFCMemoryBench: Evaluating Long-Term Memory of LLM-Based Agents in BIM Information Retrieval
  8. 2024RAGLAB: A Modular and Research-Oriented Unified Framework for Retrieval-Augmented Generation