# Back to Basics: Let Conversational Agents Remember with Just Retrieval and Generation

arXiv: https://arxiv.org/abs/2604.11628 · HTML: https://mlsanity.com/paper/2604.11628/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 30 of its 60 references that are in the map.
- First posted on arXiv 13 Apr 2026.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 30 of 30)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [MemGPT: Towards LLMs as Operating Systems](https://mlsanity.com/paper/2310.08560/) (2023)
- [Unsupervised Dense Information Retrieval with Contrastive Learning](https://mlsanity.com/paper/2112.09118/) (2021)
- [A-MEM: Agentic Memory for LLM Agents](https://mlsanity.com/paper/2502.12110/) (2025)
- [Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory](https://mlsanity.com/paper/2504.19413/) (2025)
- [LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory](https://mlsanity.com/paper/2410.10813/) (2024)
- [Zep: A Temporal Knowledge Graph Architecture for Agent Memory](https://mlsanity.com/paper/2501.13956/) (2025)
- [Memory in the Age of AI Agents](https://mlsanity.com/paper/2512.13564/) (2025)
- [From RAG to Memory: Non-Parametric Continual Learning for Large Language Models](https://mlsanity.com/paper/2502.14802/) (2025)
- [LightMem: Lightweight and Efficient Memory-Augmented Generation](https://mlsanity.com/paper/2510.18866/) (2025)
- [On Memory Construction and Retrieval for Personalized Conversational Agents](https://mlsanity.com/paper/2502.05589/) (2025)
- [MemoChat: Tuning LLMs to Use Memos for Consistent Long-Range Open-Domain Conversation](https://mlsanity.com/paper/2308.08239/) (2023)
- [MemEvolve: Meta-Evolution of Agent Memory Systems](https://mlsanity.com/paper/2512.18746/) (2025)
- [Cartridges: Lightweight and general-purpose long context representations via self-study](https://mlsanity.com/paper/2506.06266/) (2025)
- [Hierarchical Memory for High-Efficiency Long-Term Reasoning in LLM Agents](https://mlsanity.com/paper/2507.22925/) (2025)
- [MAGMA: A Multi-Graph based Agentic Memory Architecture for AI Agents](https://mlsanity.com/paper/2601.03236/) (2026)
- [Memory Layers at Scale](https://mlsanity.com/paper/2412.09764/) (2024)
- [From LLM to Conversational Agent: A Memory Enhanced Architecture with Fine-Tuning of Large Language Models](https://mlsanity.com/paper/2401.02777/) (2024)
- [TiMem: Temporal-Hierarchical Memory Consolidation for Long-Horizon Conversational Agents](https://mlsanity.com/paper/2601.02845/) (2026)
- [Fast KV Compaction via Attention Matching](https://mlsanity.com/paper/2602.16284/) (2026)
- [RCR-Router: Efficient Role-Aware Context Routing for Multi-Agent LLM Systems with Structured Memory](https://mlsanity.com/paper/2508.04903/) (2025)
- [Personalized Large Language Model Assistant with Evolving Conditional Memory](https://mlsanity.com/paper/2312.17257/) (2023)
- [A Simple Yet Strong Baseline for Long-Term Conversational Memory of LLM Agents](https://mlsanity.com/paper/2511.17208/) (2025)
- [Mem-T: Densifying Rewards for Long-Horizon Memory Agents](https://mlsanity.com/paper/2601.23014/) (2026)
- [MemGuide: Intent-Driven Memory Selection for Goal-Oriented Multi-Session LLM Agents](https://mlsanity.com/paper/2505.20231/) (2025)
- [MemoTime: Memory-Augmented Temporal Knowledge Graph Enhanced Large Language Model Reasoning](https://mlsanity.com/paper/2510.13614/) (2025)
- [RAG Meets Temporal Graphs: Time-Sensitive Modeling and Retrieval for Evolving Knowledge](https://mlsanity.com/paper/2510.13590/) (2025)
- [ARC: Active and Reflection-driven Context Management for Long-Horizon Information Seeking Agents](https://mlsanity.com/paper/2601.12030/) (2026)
- [A Game-Theoretic Spatio-Temporal Reinforcement Learning Framework for Collaborative Public Resource Allocation](https://mlsanity.com/paper/2510.26184/) (2025)

## Abstract (arXiv)

Existing conversational memory systems rely on complex hierarchical summarization or reinforcement learning to manage long-term dialogue history, yet remain vulnerable to context dilution as conversations grow. In this work, we offer a different perspective: the primary bottleneck may lie not in memory architecture, but in the textitSignal Sparsity Effect within the latent knowledge manifold. Through controlled experiments, we identify two key phenomena: textitDecisive Evidence Sparsity, where relevant signals become increasingly isolated with longer sessions, leading to sharp degradation in aggregation-based methods; and textitDual-Level Redundancy, where both inter-session interference and intra-session conversational filler introduce large amounts of non-informative content, hindering effective generation. Motivated by these insights, we propose method, a minimalist framework that brings conversational memory back to basics, relying solely on retrieval and generation via Turn Isolation Retrieval (TIR) and Query-Driven Pruning (QDP). TIR replaces global aggregation with a max-activation strategy to capture turn-level signals, while QDP removes redundant sessions and conversational filler to construct a compact, high-density evidence set. Extensive experiments on multiple benchmarks demonstrate that method achieves robust performance across diverse settings, consistently outperforming strong baselines while maintaining high efficiency in tokens and latency, establishing a new minimalist baseline for conversational memory.