# Beyond Similarity: Trustworthy Memory Search for Personal AI Agents

arXiv: https://arxiv.org/abs/2606.06054 · HTML: https://mlsanity.com/paper/2606.06054/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Placed in Agent memory systems, one of 16 directions in LLM agents and tool use.
- Builds on 30 of its 35 references that are in the map.
- First posted on arXiv 4 Jun 2026.

## Placed in

- LLM agents and tool use / [Agent memory systems](https://mlsanity.com/agents/agent-memory-systems/)

## Builds on (most cited first, 30 of 30)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions](https://mlsanity.com/paper/2311.05232/) (2023)
- [From Local to Global: A Graph RAG Approach to Query-Focused Summarization](https://mlsanity.com/paper/2404.16130/) (2024)
- [A-MEM: Agentic Memory for LLM Agents](https://mlsanity.com/paper/2502.12110/) (2025)
- [OpenAI GPT-5 System Card](https://mlsanity.com/paper/2601.03267/) (2025)
- [Evaluating Very Long-Term Conversational Memory of LLM Agents](https://mlsanity.com/paper/2402.17753/) (2024)
- [Mem0: Building Production-Ready AI Agents with Scalable Long-Term Memory](https://mlsanity.com/paper/2504.19413/) (2025)
- [RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval](https://mlsanity.com/paper/2401.18059/) (2024)
- [Search-o1: Agentic Search-Enhanced Large Reasoning Models](https://mlsanity.com/paper/2501.05366/) (2025)
- [Zep: A Temporal Knowledge Graph Architecture for Agent Memory](https://mlsanity.com/paper/2501.13956/) (2025)
- [Memory in the Age of AI Agents](https://mlsanity.com/paper/2512.13564/) (2025)
- [Evaluating Memory in LLM Agents via Incremental Multi-Turn Interactions](https://mlsanity.com/paper/2507.05257/) (2025)
- [MIRIX: Multi-Agent Memory System for LLM-Based Agents](https://mlsanity.com/paper/2507.07957/) (2025)
- [AI Agents vs. Agentic AI: A Conceptual Taxonomy, Applications and Challenges](https://mlsanity.com/paper/2505.10468/) (2025)
- [Do LLMs Recognize Your Preferences? Evaluating Personalized Preference Following in LLMs](https://mlsanity.com/paper/2502.09597/) (2025)
- [Seven Failure Points When Engineering a Retrieval Augmented Generation System](https://mlsanity.com/paper/2401.05856/) (2024)
- [MemOS: A Memory OS for AI System](https://mlsanity.com/paper/2507.03724/) (2025)
- [A Survey on Trustworthy LLM Agents: Threats and Countermeasures](https://mlsanity.com/paper/2503.09648/) (2025)
- [A Survey of Personalized Large Language Models: Progress and Future Directions](https://mlsanity.com/paper/2502.11528/) (2025)
- [OP-Bench: Benchmarking Over-Personalization for Memory-Augmented Personalized Conversational Agents](https://mlsanity.com/paper/2601.13722/) (2026)
- [PersistBench: When Should Long-Term Memories Be Forgotten by LLMs?](https://mlsanity.com/paper/2602.01146/) (2026)
- [HorizonBench: Long-Horizon Personalization with Evolving Preferences](https://mlsanity.com/paper/2604.17283/) (2026)
- [BenchPreS: A Benchmark for Context-Aware Personalized Preference Selectivity of Persistent-Memory LLMs](https://mlsanity.com/paper/2603.16557/) (2026)
- [When Personalization Legitimizes Risks: Uncovering Safety Vulnerabilities in Personalized Dialogue Agents](https://mlsanity.com/paper/2601.17887/) (2026)
- [Remembering More, Risking More: Longitudinal Safety Risks in Memory-Equipped LLM Agents](https://mlsanity.com/paper/2605.17830/) (2026)
- [Memory-Induced Tool-Drift in LLM Agents](https://mlsanity.com/paper/2605.24941/) (2026)
- [When Should Models Change Their Minds? Contextual Belief Management in Large Language Models](https://mlsanity.com/paper/2605.30219/) (2026)

## Built on by (most cited first, 12 of 12)

- [Mi-Memory: A Lifecycle Memory Framework for Personal AI](https://mlsanity.com/paper/2607.18975/) (2026)
- [SuperLocalMemory 4.0: The Governed Memory Operating System for AI Agents](https://mlsanity.com/paper/2608.08253/) (2026)
- [The Retriever Should Remember: Experience-Amortized Reranking for Long-Term Agent Memory](https://mlsanity.com/paper/2608.22767/) (2026)
- [DreamBench-SWE: A Multi-Session Memory-Hygiene Benchmark for Software Agents](https://mlsanity.com/paper/2608.20664/) (2026)
- [MeClear: Cooperative Game-Theoretic Attribution and Risk-Aware Memory Clearance for Long-Horizon LLM Agents](https://mlsanity.com/paper/2609.09115/) (2026)
- [Emergi-PersonaOS: A Persona Agent Operating System for Situational Adaptation and Controllable Evolution](https://mlsanity.com/paper/2609.27417/) (2026)
- [PairPref: When Should Memory Guide the Answer? A Benchmark for Contextual Preference Use](https://mlsanity.com/paper/2609.34526/) (2026)
- [Memadapter: Counterfactual Adaptation Against Memory-induced Sycophancy](https://mlsanity.com/paper/2610.05162/) (2026)
- [The Right Memory in the Wrong Context: Verifying Retrieval Admissibility in Long-Term Agent Memory](https://mlsanity.com/paper/2610.07309/) (2026)
- [Understanding and Mitigating Inference-Time Overreliance Using Agentic Memory](https://mlsanity.com/paper/2610.07311/) (2026)
- [Defense-in-Depth for LLMs: Evaluating Memory Gates Against Activation-Induced and Memory-Induced Sycophancy](https://mlsanity.com/paper/2610.07403/) (2026)
- [What to Admit and How to Present: Governing Persistent Memory in LLM Agents](https://mlsanity.com/paper/2610.11188/) (2026)

## Abstract (arXiv)

Personal AI agents increasingly rely on long-term memory to provide persistent personalization across sessions. However, existing memory pipelines are largely driven by semantic similarity: memory data close to the current query is retrieved and injected into the model context. This creates a critical trustworthiness gap, since a semantically related memory may still be contextually inappropriate, leading to threats such as cross-domain leakage, sycophancy, tool-call drift, or memory-induced jailbreaks. In this paper, we study memory search as a trust boundary in personal AI agents. We evaluate representative agentic memory frameworks, including A-Mem, Mem0, and MemOS, together with OpenClaw, a real-world personal-agent environment with persistent state and tool-use capability. Our results show that long-term memory is not merely a utility layer, but a durable control channel that can reshape how agents interpret tasks and execute actions, leaving them highly susceptible to the aforementioned threats. To mitigate these vulnerabilities, we propose MemGate, a lightweight and deployable memory plug-in for trustworthy memory search, with only 9M parameters and a 35.1MB footprint. MemGate is inserted between the vector memory store and the backbone LLM, requiring no LLM modification, memory-database rewriting, or inference-time LLM judge. It applies a query-conditioned neural gate to candidate memory representations, turning raw similarity search into task-conditioned memory admission. Across multiple mainstream memory frameworks, real-world agent settings, and diverse LLM backbones, MemGate reduces memory-induced threats while preserving long-term memory utility.