# Evoking User Memory: Personalizing LLM via Recollection-Familiarity Adaptive Retrieval

arXiv: https://arxiv.org/abs/2603.09250 · HTML: https://mlsanity.com/paper/2603.09250/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Mar 2026.

## Built on by (most cited first, 8 of 8)

- [ElasticMem: Latent Memory as a Learnable Resource for LLM Agents](https://mlsanity.com/paper/2605.30690/) (2026)
- [RippleMem: From Isolated Retrieval to Associative Recollection for Long-Term Agent Memory](https://mlsanity.com/paper/2608.13334/) (2026)
- [Learning How and What to Memorize: Cognition-Inspired Two-Stage Optimization for Evolving Memory](https://mlsanity.com/paper/2605.00702/) (2026)
- [AdaMem: Learning What to Remember with Adaptive Memory Policies for Personalized Agents](https://mlsanity.com/paper/2606.21144/) (2026)
- [Learning What to Share and What to Personalize: Hierarchical Strategy Co-Evolution for Agent Memory](https://mlsanity.com/paper/2608.25329/) (2026)
- [UniDial-EvalKit: A Unified Toolkit for Evaluating Multi-Faceted Conversational Abilities](https://mlsanity.com/paper/2603.23160/) (2026)
- [MGRetrieval: Memory-Guided Reflective Retrieval for Long-Term Dialogue Agents](https://mlsanity.com/paper/2605.27437/) (2026)
- [ThinkFlow: Self-Evolving Probabilistic Latent Memory for Lifelong Conversational Agents](https://mlsanity.com/paper/2609.17010/) (2026)

## Abstract (arXiv)

Personalized large language models (LLMs) rely on memory retrieval to incorporate user-specific histories, preferences, and contexts. Existing approaches either overload the LLM by feeding all the user's past memory into the prompt, which is costly and unscalable, or simplify retrieval into a one-shot similarity search, which captures only surface matches. Cognitive science, however, shows that human memory operates through a dual process: Familiarity, offering fast but coarse recognition, and Recollection, enabling deliberate, chain-like reconstruction for deeply recovering episodic content. Current systems lack both the ability to perform recollection retrieval and mechanisms to adaptively switch between the dual retrieval paths, leading to either insufficient recall or the inclusion of noise. To address this, we propose RF-Mem (Recollection-Familiarity Memory Retrieval), a familiarity uncertainty-guided dual-path memory retriever. RF-Mem measures the familiarity signal through the mean score and entropy. High familiarity leads to the direct top-K Familiarity retrieval path, while low familiarity activates the Recollection path. In the Recollection path, the system clusters candidate memories and applies alpha-mix with the query to iteratively expand evidence in embedding space, simulating deliberate contextual reconstruction. This design embeds human-like dual-process recognition into the retriever, avoiding full-context overhead and enabling scalable, adaptive personalization. Experiments across three benchmarks and corpus scales demonstrate that RF-Mem consistently outperforms both one-shot retrieval and full-context reasoning under fixed budget and latency constraints. Our code can be found in the Reproducibility Statement.