# Beyond RAG for Agent Memory: Retrieval by Decoupling and Aggregation

arXiv: https://arxiv.org/abs/2602.02007 · HTML: https://mlsanity.com/paper/2602.02007/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in Agentic RAG, agents and memory, one of 15 directions in Retrieval-augmented generation.
- Builds on 25 of its 34 references that are in the map.
- First posted on arXiv 2 Feb 2026.

## Placed in

- Retrieval-augmented generation / [Agentic RAG, agents and memory](https://mlsanity.com/rag/agentic-rag-agents-and-memory/)
- LLM agents and tool use / [Agent memory systems](https://mlsanity.com/agents/agent-memory-systems/)

## Builds on (most cited first, 25 of 25)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Retrieval-Augmented Generation for Large Language Models: A Survey](https://mlsanity.com/paper/2312.10997/) (2023)
- [MemGPT: Towards LLMs as Operating Systems](https://mlsanity.com/paper/2310.08560/) (2023)
- [A-MEM: Agentic Memory for LLM Agents](https://mlsanity.com/paper/2502.12110/) (2025)
- [Evaluating Very Long-Term Conversational Memory of LLM Agents](https://mlsanity.com/paper/2402.17753/) (2024)
- [MemoryBank: Enhancing Large Language Models with Long-Term Memory](https://mlsanity.com/paper/2305.10250/) (2023)
- [RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval](https://mlsanity.com/paper/2401.18059/) (2024)
- [LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression](https://mlsanity.com/paper/2310.06839/) (2023)
- [Zep: A Temporal Knowledge Graph Architecture for Agent Memory](https://mlsanity.com/paper/2501.13956/) (2025)
- [Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents](https://mlsanity.com/paper/2304.09542/) (2023)
- [LLMLingua-2: Data Distillation for Efficient and Faithful Task-Agnostic Prompt Compression](https://mlsanity.com/paper/2403.12968/) (2024)
- [Memory in the Age of AI Agents](https://mlsanity.com/paper/2512.13564/) (2025)
- [Beyond Goldfish Memory: Long-Term Open-Domain Conversation](https://mlsanity.com/paper/2107.07567/) (2021)
- [LightMem: Lightweight and Efficient Memory-Augmented Generation](https://mlsanity.com/paper/2510.18866/) (2025)
- [Memory OS of AI Agent](https://mlsanity.com/paper/2506.06326/) (2025)
- [In Prospect and Retrospect: Reflective Memory Management for Long-term Personalized Dialogue Agents](https://mlsanity.com/paper/2503.08026/) (2025)
- [EverMemOS: A Self-Organizing Memory Operating System for Structured Long-Horizon Reasoning](https://mlsanity.com/paper/2601.02163/) (2026)
- [Conversation Chronicles: Towards Diverse Temporal and Relational Dynamics in Multi-Session Conversations](https://mlsanity.com/paper/2310.13420/) (2023)
- [AI PERSONA: Towards Life-long Personalization of LLMs](https://mlsanity.com/paper/2412.13103/) (2024)
- [CAM: A Constructivist View of Agentic Memory for LLM-Based Reading Comprehension](https://mlsanity.com/paper/2510.05520/) (2025)
- [LeanRAG: Knowledge-Graph-Based Generation with Semantic Aggregation and Hierarchical Retrieval](https://mlsanity.com/paper/2508.10391/) (2025)

## Built on by (most cited first, 19 of 19)

- [Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering](https://mlsanity.com/paper/2604.08224/) (2026)
- [Memory as a Controlled Process: Learned Adaptive Memory Management for LLM Agents](https://mlsanity.com/paper/2607.13591/) (2026)
- [MemTrace: Tracing and Attributing Errors in Large Language Model Memory Systems](https://mlsanity.com/paper/2605.28732/) (2026)
- [Toward a Theory of Hierarchical Memory for Language Agents](https://mlsanity.com/paper/2603.21564/) (2026)
- [Rethinking How to Remember: Beyond Atomic Facts in Lifelong LLM Agent Memory](https://mlsanity.com/paper/2605.19952/) (2026)
- [CUADebug: Diagnosing and Repairing Computer-Use Agent Failures](https://mlsanity.com/paper/2608.02643/) (2026)
- [DimMem: Dimensional Structuring for Efficient Long-Term Agent Memory](https://mlsanity.com/paper/2605.15759/) (2026)
- [DOVA: Deliberation-First Multi-Agent Orchestration for Autonomous Research Automation](https://mlsanity.com/paper/2603.13327/) (2026)
- [Keep It InMind: Benchmarking the Implicit-Association Blind Spot in Agent Memory](https://mlsanity.com/paper/2607.24368/) (2026)
- [MemoryCPT: An End-to-End Agent Memory Framework for Cost-Performance Trade-off](https://mlsanity.com/paper/2608.04843/) (2026)
- [Personalization Meets Safety:Mechanisms,Risks,and Mitigations in Personalized LLMs](https://mlsanity.com/paper/2606.09038/) (2026)
- [Seek to Segment: Active Perception for Panoramic Referring Segmentation](https://mlsanity.com/paper/2607.02497/) (2026)
- [Memory-Orchestrated Semantic System (MOSS): An Auditable Agentic Memory Architecture](https://mlsanity.com/paper/2607.04391/) (2026)
- [Memory in the Loop: In-Process Retrieval as Extended Working Memory for Language Agents](https://mlsanity.com/paper/2607.05690/) (2026)
- [MemForest: Efficient Agent Memory Management via EventTree Partitioning and Progressive Merging](https://mlsanity.com/paper/2609.08273/) (2026)
- [Graph-Based Personalized Memory for LLM Agents: Representation, Evolution, Retrieval, and Evaluation](https://mlsanity.com/paper/2609.08599/) (2026)
- [The Missing Complement: State-Conditioned Minimal Sufficient Evidence for Coding Agents](https://mlsanity.com/paper/2609.20050/) (2026)
- [Test-Time Memory Construction](https://mlsanity.com/paper/2610.13348/) (2026)
- [Recall What Is Missing: Composing Evidence from Long-Term Agent Memory](https://mlsanity.com/paper/2610.15642/) (2026)

## Abstract (arXiv)

Standard Retrieval Augmented Generation (RAG) is poorly matched to agent memory. Unlike large heterogeneous corpora, agent memory forms a bounded and coherent interaction stream in which many spans are highly correlated or near duplicates. As a result, flat top-k similarity retrieval often returns redundant context, while summary-centric hierarchies can blur the subtle details that distinguish one candidate from another. We argue that agent memory should follow the principle of decoupling before aggregation: the system should first isolate reusable facts, updates, and distinguishing details from similar histories, and only then organise them for efficient retrieval. Based on this principle, we propose xMemory, which constructs a revisable hierarchical memory structure from original messages to segments, memory components, and groups. xMemory segments interaction history into local events, decouples each segment into memory components, aggregates related components into high-level groups using a sparsity--semantic faithfulness objective, and maintains this structure incrementally as memory evolves. At inference time, xMemory retrieves top-down, first selecting a compact backbone of complementary groups and components, and then expanding to segments and raw messages only when additional evidence reduces the reader's uncertainty. Experiments on LoCoMo and PerLTQA across diverse open source and closed source LLMs show consistent gains in answer quality and inference token efficiency, supported by analyses of redundancy, evidence density, and coverage.