# From Single to Multi-Granularity: Toward Long-Term Memory Association and Selection of Conversational Agents

arXiv: https://arxiv.org/abs/2505.19549 · HTML: https://mlsanity.com/paper/2505.19549/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- Placed in Agent memory systems, one of 16 directions in LLM agents and tool use.
- Builds on 52 of its 61 references that are in the map.
- First posted on arXiv 26 May 2025.

## Placed in

- LLM agents and tool use / [Agent memory systems](https://mlsanity.com/agents/agent-memory-systems/)

## Builds on (most cited first, 30 of 52)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Dense Passage Retrieval for Open-Domain Question Answering](https://mlsanity.com/paper/2004.04906/) (2020)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023)
- [From Local to Global: A Graph RAG Approach to Query-Focused Summarization](https://mlsanity.com/paper/2404.16130/) (2024)
- [MemGPT: Towards LLMs as Operating Systems](https://mlsanity.com/paper/2310.08560/) (2023)
- [Text Embeddings by Weakly-Supervised Contrastive Pre-training](https://mlsanity.com/paper/2212.03533/) (2022)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [A-MEM: Agentic Memory for LLM Agents](https://mlsanity.com/paper/2502.12110/) (2025)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [ChatDev: Communicative Agents for Software Development](https://mlsanity.com/paper/2307.07924/) (2023)
- [Evaluating Very Long-Term Conversational Memory of LLM Agents](https://mlsanity.com/paper/2402.17753/) (2024)
- [Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory](https://mlsanity.com/paper/2504.19413/) (2025)
- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)
- [LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory](https://mlsanity.com/paper/2410.10813/) (2024)
- [A Survey on the Memory Mechanism of Large Language Model based Agents](https://mlsanity.com/paper/2404.13501/) (2024)
- [MemoryBank: Enhancing Large Language Models with Long-Term Memory](https://mlsanity.com/paper/2305.10250/) (2023)
- [RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval](https://mlsanity.com/paper/2401.18059/) (2024)
- [ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction](https://mlsanity.com/paper/2112.01488/) (2021)
- [Zep: A Temporal Knowledge Graph Architecture for Agent Memory](https://mlsanity.com/paper/2501.13956/) (2025)
- [Beyond Goldfish Memory: Long-Term Open-Domain Conversation](https://mlsanity.com/paper/2107.07567/) (2021)
- [Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Security](https://mlsanity.com/paper/2401.05459/) (2024)
- [RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation](https://mlsanity.com/paper/2310.04408/) (2023)
- [From RAG to Memory: Non-Parametric Continual Learning for Large Language Models](https://mlsanity.com/paper/2502.14802/) (2025)
- [G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering](https://mlsanity.com/paper/2402.07630/) (2024)
- [A Human-Inspired Reading Agent with Gist Memory of Very Long Contexts](https://mlsanity.com/paper/2402.09727/) (2024)
- [From Human Memory to AI Memory: A Survey on Memory Mechanisms in the Era of LLMs](https://mlsanity.com/paper/2504.15965/) (2025)
- [Hello Again! LLM-powered Personalized Agent for Long-term Dialogue](https://mlsanity.com/paper/2406.05925/) (2024)
- [Graph Chain-of-Thought: Augmenting Large Language Models by Reasoning on Graphs](https://mlsanity.com/paper/2404.07103/) (2024)

## Built on by (most cited first, 29 of 29)

- [SGMem: Sentence Graph Memory for Long-Term Conversational Agents](https://mlsanity.com/paper/2509.21212/) (2025)
- [PERMA: Benchmarking Personalized Memory Agents via Event-Driven Preference and Realistic Task Environments](https://mlsanity.com/paper/2603.23231/) (2026)
- [AI Meets Brain: Memory Systems from Cognitive Neuroscience to Autonomous Agents](https://mlsanity.com/paper/2512.23343/) (2025)
- [Memory in the LLM Era: Modular Architectures and Strategies in a Unified Framework](https://mlsanity.com/paper/2604.01707/) (2026)
- [Choosing How to Remember: Adaptive Memory Structures for LLM Agents](https://mlsanity.com/paper/2602.14038/) (2026)
- [RecMem: Recurrence-based Memory Consolidation for Efficient and Effective Long-Running LLM Agents](https://mlsanity.com/paper/2605.16045/) (2026)
- [GeoArena: Evaluating Open-World Geographic Reasoning in Large Vision-Language Models](https://mlsanity.com/paper/2509.04334/) (2025)
- [H-Mem: A Novel Memory Mechanism for Evolving and Retrieving Agent Memory via a Hybrid Structure](https://mlsanity.com/paper/2605.15701/) (2026)
- [Oblivion: Self-Adaptive Agentic Memory Control through Decay-Driven Activation](https://mlsanity.com/paper/2604.00131/) (2026)
- [RippleMem: From Isolated Retrieval to Associative Recollection for Long-Term Agent Memory](https://mlsanity.com/paper/2608.13334/) (2026)
- [Learning a Single Token to Replace Long System Prompts in LLMs](https://mlsanity.com/paper/2511.23271/) (2025)
- [Learning How and What to Memorize: Cognition-Inspired Two-Stage Optimization for Evolving Memory](https://mlsanity.com/paper/2605.00702/) (2026)
- [STRIDE: Strategic Iterative Decision-Making for Retrieval-Augmented Multi-Hop Question Answering](https://mlsanity.com/paper/2604.17405/) (2026)
- [DeferMem: Query-Time Evidence Distillation via Reinforcement Learning for Long-Term Agent Memory](https://mlsanity.com/paper/2605.22411/) (2026)
- [Entropy-KL Divergence-based Token Masking: A Novel Approach for Selective Fine-tuning of Large Language Models](https://mlsanity.com/paper/2605.29303/) (2026)
- [OPD-Evolver: Cultivating Holistic Agent Evolver via On-Policy Distillation](https://mlsanity.com/paper/2606.17628/) (2026)
- [MemSIF: From Structured Interactions to Dual-Track Fact Memory for LLM Agents](https://mlsanity.com/paper/2608.01742/) (2026)
- [GraphER: An Efficient Graph-Based Enrichment and Reranking Method for Retrieval-Augmented Generation](https://mlsanity.com/paper/2603.24925/) (2026)
- [Task-Adaptive Retrieval over Agentic Multi-Modal Web Histories via Learned Graph Memory](https://mlsanity.com/paper/2604.07863/) (2026)
- [Memory-Augmented LLM-based Multi-Agent System for Automated Feature Generation on Tabular Data](https://mlsanity.com/paper/2604.20261/) (2026)
- [EdgeMem: LLM-Free Agent Memory Construction and Retrieval via Evidence-Preserving Multi-Anchor Hypergraph](https://mlsanity.com/paper/2609.05553/) (2026)
- [Where to Look and What to Use: Retrieve-Localize-Generate for Long-Term Conversational Memory Question Answering](https://mlsanity.com/paper/2609.07093/) (2026)
- [CreaMem: A Scene-Aware Memory Architecture for Personalized Agents](https://mlsanity.com/paper/2609.08550/) (2026)
- [AutoViewMem: Self-Configuring Orthogonal Views for Conversational Long-Term Memory](https://mlsanity.com/paper/2609.21940/) (2026)
- [EP-Mem: Elastic Privacy Memory for Social Relationship-Aware LLM Agents](https://mlsanity.com/paper/2609.35233/) (2026)
- [MemCodex: Self-Programming Hierarchical Memory for Language Agents](https://mlsanity.com/paper/2609.39765/) (2026)
- [Memory Type Varies: Empowering LLM Agents for Long-Term Memory with Diverse Strategies](https://mlsanity.com/paper/2610.11573/) (2026)
- [Temporal Evidence Forests for Long-Horizon Multi-Party Conversational Memory](https://mlsanity.com/paper/2610.12874/) (2026)
- [Reflect After Failure: Brain-Inspired Self-Evolving Agents for Embodied Spatial Intelligence](https://mlsanity.com/paper/2610.20312/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) have recently been widely adopted in conversational agents. However, the increasingly long interactions between users and agents accumulate extensive dialogue records, making it difficult for LLMs with limited context windows to maintain a coherent long-term dialogue memory and deliver personalized responses. While retrieval-augmented memory systems have emerged to address this issue, existing methods often depend on single-granularity memory segmentation and retrieval. This approach falls short in capturing deep memory connections, leading to partial retrieval of useful information or substantial noise, resulting in suboptimal performance. To tackle these limits, we propose MemGAS, a framework that enhances memory consolidation by constructing multi-granularity association, adaptive selection, and retrieval. MemGAS is based on multi-granularity memory units and employs Gaussian Mixture Models to cluster and associate new memories with historical ones. An entropy-based router adaptively selects optimal granularity by evaluating query relevance distributions and balancing information completeness and noise. Retrieved memories are further refined via LLM-based filtering. Experiments on four long-term memory benchmarks demonstrate that MemGAS outperforms state-of-the-art methods on both question answer and retrieval tasks, achieving superior performance across different query types and top-K settings. footnotehttps://github.com/quqxui/MemGAS