# Memory Retrieval for Changing Preferences

arXiv: https://arxiv.org/abs/2606.02976 · HTML: https://mlsanity.com/paper/2606.02976/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 34 references that are in the map.
- First posted on arXiv 2 Jun 2026.

## Builds on (most cited first, 25 of 25)

- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [Large Language Models Can Be Easily Distracted by Irrelevant Context](https://mlsanity.com/paper/2302.00093/) (2023)
- [Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory](https://mlsanity.com/paper/2504.19413/) (2025)
- [LongMemEval: Benchmarking Chat Assistants on Long-Term Interactive Memory](https://mlsanity.com/paper/2410.10813/) (2024)
- [MemoryBank: Enhancing Large Language Models with Long-Term Memory](https://mlsanity.com/paper/2305.10250/) (2023)
- [LaMP: When Large Language Models Meet Personalization](https://mlsanity.com/paper/2304.11406/) (2023)
- [LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens](https://mlsanity.com/paper/2402.13753/) (2024)
- [RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation](https://mlsanity.com/paper/2310.04408/) (2023)
- [Know Me, Respond to Me: Benchmarking LLMs for Dynamic User Profiling and Personalized Responses at Scale](https://mlsanity.com/paper/2504.14225/) (2025)
- [Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMs](https://mlsanity.com/paper/2502.09597/) (2025)
- [Hello Again! LLM-powered Personalized Agent for Long-term Dialogue](https://mlsanity.com/paper/2406.05925/) (2024)
- [In Prospect and Retrospect: Reflective Memory Management for Long-term Personalized Dialogue Agents](https://mlsanity.com/paper/2503.08026/) (2025)
- [MemBench: Towards More Comprehensive Evaluation on the Memory of LLM-based Agents](https://mlsanity.com/paper/2506.21605/) (2025)
- [Panoptic Scene Graph Generation with Semantics-Prototype Learning](https://mlsanity.com/paper/2307.15567/) (2023)
- [Treble Counterfactual VLMs: A Causal Approach to Hallucination](https://mlsanity.com/paper/2503.06169/) (2025)
- [The Distracting Effect: Understanding Irrelevant Passages in RAG](https://mlsanity.com/paper/2505.06914/) (2025)
- [DPU: Dynamic Prototype Updating for Multimodal Out-of-Distribution Detection](https://mlsanity.com/paper/2411.08227/) (2024)
- [Secure On-Device Video OOD Detection Without Backpropagation](https://mlsanity.com/paper/2503.06166/) (2025)
- [A Personalized Conversational Benchmark: Towards Simulating Personalized Conversations](https://mlsanity.com/paper/2505.14106/) (2025)
- [R4ec: A Reasoning, Reflection, and Refinement Framework for Recommendation Systems](https://mlsanity.com/paper/2507.17249/) (2025)
- [The Autonomy Tax: Defense Training Breaks LLM Agents](https://mlsanity.com/paper/2603.19423/) (2026)
- [Pointwise Mutual Information as a Performance Gauge for Retrieval-Augmented Generation](https://mlsanity.com/paper/2411.07773/) (2024)
- [Impact-driven Context Filtering For Cross-file Code Completion](https://mlsanity.com/paper/2508.05970/) (2025)
- [Self-Evolving LLM Memory Extraction Across Heterogeneous Tasks](https://mlsanity.com/paper/2604.11610/) (2026)

## Built on by (most cited first, 3 of 3)

- [RaMem: Contextual Reinstatement for Long-term Agentic Memory](https://mlsanity.com/paper/2606.22844/) (2026)
- [SynCo: Data Synthesis Co-Training for Self-Evolving LLMs via Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2610.11345/) (2026)
- [Controlled Decoding Attacks on Black-Box LLMs](https://mlsanity.com/paper/2609.36956/) (2026)

## Abstract (arXiv)

Long-context dialogue systems must decide both when to access memory and which parts of the interaction history are relevant. Existing approaches typically rely on heuristic retrieval signals or always-on memory usage, failing to account for the changing and potentially inconsistent nature of user preferences. In this work, we propose a unified framework for memory access and selection based on changing preferences. We formulate personalized memory retrieval as identifying which historical turns provide evidence about a user's latent preference state, rather than relying on surface-level semantic similarity. To this end, we quantify the utility of each memory turn using a Bayes factor, defined as the improvement in the model's likelihood of the reference response when the turn is included in context. This provides a principled measure of evidence strength and a unified signal for both memory access and selection. By framing memory retrieval as utility estimation, the model learns to identify salient turns and regulate memory usage based on expected utility. Experiments on four heterogeneous memory benchmarks show that our approach outperforms existing embedding-based retrieval on long-context, preference-intensive tasks where modeling changing preferences is essential, while remaining competitive in low-density regimes where semantic similarity suffices.