# What Deserves Memory: Adaptive Memory Distillation for LLM Agents

arXiv: https://arxiv.org/abs/2508.03341 · HTML: https://mlsanity.com/paper/2508.03341/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- Placed in Agent memory systems, one of 16 directions in LLM agents and tool use.
- Builds on 10 of its 25 references that are in the map.
- First posted on arXiv 5 Aug 2025.

## Placed in

- LLM agents and tool use / [Agent memory systems](https://mlsanity.com/agents/agent-memory-systems/)

## Builds on (most cited first, 10 of 10)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [MemGPT: Towards LLMs as Operating Systems](https://mlsanity.com/paper/2310.08560/) (2023)
- [A-MEM: Agentic Memory for LLM Agents](https://mlsanity.com/paper/2502.12110/) (2025)
- [Evaluating Very Long-Term Conversational Memory of LLM Agents](https://mlsanity.com/paper/2402.17753/) (2024)
- [LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory](https://mlsanity.com/paper/2410.10813/) (2024)
- [Zep: A Temporal Knowledge Graph Architecture for Agent Memory](https://mlsanity.com/paper/2501.13956/) (2025)
- [Hello Again! LLM-powered Personalized Agent for Long-term Dialogue](https://mlsanity.com/paper/2406.05925/) (2024)
- [AriGraph: Learning Knowledge Graph World Models with Episodic Memory for LLM Agents](https://mlsanity.com/paper/2407.04363/) (2024)
- [Emotional RAG: Enhancing Role-Playing Agents through Emotional Retrieval](https://mlsanity.com/paper/2410.23041/) (2024)

## Built on by (most cited first, 30 of 48)

- [Memory in the Age of AI Agents](https://mlsanity.com/paper/2512.13564/) (2025)
- [LightMem: Lightweight and Efficient Memory-Augmented Generation](https://mlsanity.com/paper/2510.18866/) (2025)
- [Mem-α: Learning Memory Construction via Reinforcement Learning](https://mlsanity.com/paper/2509.25911/) (2025)
- [EverMemOS: A Self-Organizing Memory Operating System for Structured Long-Horizon Reasoning](https://mlsanity.com/paper/2601.02163/) (2026)
- [A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agents](https://mlsanity.com/paper/2601.12538/) (2026)
- [A Survey of Agent Memory in the Second Half: Towards Self-Evolving and Long-Horizon Agents](https://mlsanity.com/paper/2602.06052/) (2026)
- [TiMem: Temporal-Hierarchical Memory Consolidation for Long-Horizon Conversational Agents](https://mlsanity.com/paper/2601.02845/) (2026)
- [Graph-based Agent Memory: Taxonomy, Techniques, and Applications](https://mlsanity.com/paper/2602.05665/) (2026)
- [A Simple Yet Strong Baseline for Long-Term Conversational Memory of LLM Agents](https://mlsanity.com/paper/2511.17208/) (2025)
- [AMemGym: Interactive Memory Benchmarking for Assistants in Long-Horizon Conversations](https://mlsanity.com/paper/2603.01966/) (2026)
- [HiMem: Hierarchical Long-Term Memory for LLM Long-Horizon Agents](https://mlsanity.com/paper/2601.06377/) (2026)
- [Learning to Continually Learn via Meta-learning Agentic Memory Designs](https://mlsanity.com/paper/2602.07755/) (2026)
- [Governing Evolving Memory in LLM Agents: Risks, Mechanisms, and the Stability and Safety Governed Memory (SSGM) Framework](https://mlsanity.com/paper/2603.11768/) (2026)
- [APEX-MEM: Agentic Semi-Structured Memory with Temporal Reasoning for Long-Term Conversational AI](https://mlsanity.com/paper/2604.14362/) (2026)
- [TeleMem: Building Long-Term and Multimodal Memory for Agentic AI](https://mlsanity.com/paper/2601.06037/) (2025)
- [Toward Personalized LLM-Powered Agents: Foundations, Evaluation, and Future Directions](https://mlsanity.com/paper/2602.22680/) (2026)
- [HyperMem: Hypergraph Memory for Long-Term Conversations](https://mlsanity.com/paper/2604.08256/) (2026)
- [MEMTRACK: Evaluating Long-Term Memory and State Tracking in Multi-Platform Dynamic Agent Environments](https://mlsanity.com/paper/2510.01353/) (2025)
- [AI Meets Brain: Memory Systems from Cognitive Neuroscience to Autonomous Agents](https://mlsanity.com/paper/2512.23343/) (2025)
- [Does Memory Need Graphs? A Unified Framework and Empirical Analysis for Long-Term Dialog Memory](https://mlsanity.com/paper/2601.01280/) (2026)
- [Beyond Pipelines: A Survey of the Paradigm Shift toward Model-Native Agentic AI](https://mlsanity.com/paper/2510.16720/) (2025)
- [Remember the Decision, Not the Description: A Rate-Distortion Framework for Agent Memory](https://mlsanity.com/paper/2605.10870/) (2026)
- [EMBER: Efficient Memory via Budgeted Evidence Retention for Long-Horizon Agents](https://mlsanity.com/paper/2606.05894/) (2026)
- [RaMem: Contextual Reinstatement for Long-term Agentic Memory](https://mlsanity.com/paper/2606.22844/) (2026)
- [SmartSearch: How Ranking Beats Structure for Conversational Memory Retrieval](https://mlsanity.com/paper/2603.15599/) (2026)
- [D-Mem: A Dual-Process Memory System for LLM Agents](https://mlsanity.com/paper/2603.18631/) (2026)
- [MemQ: Integrating Q-Learning into Self-Evolving Memory Agents over Provenance DAGs](https://mlsanity.com/paper/2605.08374/) (2026)
- [Mitigating Provenance-Role Collapse in Long-Term Agents via Typed Memory Representation](https://mlsanity.com/paper/2605.25869/) (2026)
- [ES-Mem: Event Segmentation-Based Memory for Long-Term Dialogue Agents](https://mlsanity.com/paper/2601.07582/) (2026)
- [MemSifter: Offloading LLM Memory Retrieval via Outcome-Driven Proxy Reasoning](https://mlsanity.com/paper/2603.03379/) (2026)

## Abstract (arXiv)

Memory systems for LLM agents struggle to determine what information deserves retention. Existing approaches rely on predefined heuristics such as importance scores, emotional tags, or factual templates, encoding designer intuition rather than learning from the data itself. Inspired by cognitive ideas, we propose NEMORI, an adaptive memory distillation framework that casts the assessment of experience's future utility as a matter of predictability. Specifically, NEMORI comprises two cascading modules: Episodic Memory Integration transforms raw interactions into coherent narratives, and Semantic Knowledge Distillation extracts insights via prediction error. Centering on distillation, the framework remains agnostic to downstream management. Extensive experiments confirm that NEMORI achieves strong performance, efficiency, and storage reduction. Our work suggests that observing the intrinsic properties of interaction sequences offers a viable, data-driven alternative to heuristic-based memory design. Code: https://github.com/nemori-ai/nemori.