When Personalization Legitimizes Risks: Uncovering Safety Vulnerabilities in Personalized Dialogue Agents
On the map: LLM agents and tool use → Agent memory systems
Abstract (arXiv)
Long-term memory enables large language model (LLM) agents to support personalized and sustained interactions. However, most work on personalized agents prioritizes utility and user experience, treating memory as a neutral component and largely overlooking its safety implications. In this paper, we reveal intent legitimation, a previously underexplored safety failure in personalized agents, where benign personal memories bias intent inference and cause models to legitimize inherently harmful queries. To study this phenomenon, we introduce PS-Bench, a benchmark designed to identify and quantify intent legitimation in personalized interactions. Across multiple memory-augmented agent frameworks and base LLMs, personalization increases attack success rates by 15.8%--243.7% relative to stateless baselines. We further provide mechanistic evidence for intent legitimation from internal representations space, and propose a lightweight detection-reflection method that effectively reduces safety degradation. Overall, our work provides the first systematic exploration and evaluation of intent legitimation as a safety failure mode that naturally arises from benign, real-world personalization, highlighting the importance of assessing safety under long-term personal context. Our code is available at: https://github.com/MuyuenLP/PS-Bench. WARNING: This paper may contain harmful content.
Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Placed in Agent memory systems, one of 16 directions in LLM agents and tool use. Builds on 31 of its 49 references that are in the map. First posted on arXiv 25 Jan 2026.
Builds on
31 of 49 references.
Closest ancestors · 6
Same direction, newest first
- 2026OP-Bench: Benchmarking Over-Personalization for Memory-Augmented Personalized Conversational Agents20 Jan
- 2025PersonaMem-v2: Towards Personalized Intelligence via Learning Implicit User Personas and Agentic Memory7 Dec
- 2025O-Mem: Omni Memory System for Personalized, Long Horizon, Self-Evolving Agents17 Nov
- 2025A-MEM: Agentic Memory for LLM Agents17 Feb
- 2024Hello Again! LLM-powered Personalized Agent for Long-term Dialogue9 Jun
- 2024Evaluating Very Long-Term Conversational Memory of LLM Agents27 Feb
Other references · 25
- 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
- 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
- 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
- 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 828
- 2022A Survey on In-context Learningcited by 721
- 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 472
- 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 431
- 2024AgentPoison: Red-teaming LLM Agents via Poisoning Memory or Knowledge Basescited by 429
- 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 223
- 2024PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Actioncited by 171
- 2023Poisoning Retrieval Corpora by Injecting Adversarial Passagescited by 159
- 2025MemOS: A Memory OS for AI Systemcited by 138
13 more
- 2024ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teamingcited by 98
- 2024How Alignment and Jailbreak Work: Explain LLM Safety through Intermediate Hidden Statescited by 98
- 2025A Survey on Trustworthy LLM Agents: Threats and Countermeasurescited by 96
- 2024Uncovering Safety Risks of Large Language Models through Concept Activation Vectorcited by 67
- 2023Hijacking Large Language Models via Adversarial In-Context Learningcited by 45
- 2025AdaSteer: Your Aligned LLM is Inherently an Adaptive Jailbreak Defendercited by 38
- 2025Teaching Language Models to Evolve with Users: Dynamic Profile Modeling for Personalized Alignmentcited by 24
- 2020Refer, Reuse, Reduce: Generating Subsequent References in Visual and Conversational Contextscited by 22
- 2025Beware of Your Po! Measuring and Mitigating AI Safety Risks in Role-Play Fine-Tuning of LLMscited by 22
- 2024Towards Comprehensive Post Safety Alignment of Large Language Models via Safety Patchingcited by 17
- 2025Personalized Attacks of Social Engineering in Multi-turn Conversations: LLM Agents for Simulation and Detectioncited by 13
- 2025SAGE: A Generic Framework for LLM Safety Evaluationcited by 8
- 2025Evolution in Simulation: AI-Agent School with Dual Memory for High-Fidelity Educational Dynamicscited by 3
Built on by
Citing papers per year: 2026: 7.
7 papers.
- 2026A Survey on Long-Term Memory Security in LLM Agents: Attacks, Defenses, and Governance Across the Memory Lifecyclecited by 27
- 2026Beyond Similarity: Trustworthy Memory Search for Personal AI Agentscited by 12
- 2026BenchPreS: A Benchmark for Context-Aware Personalized Preference Selectivity of Persistent-Memory LLMscited by 10
- 2026TEA-Bench: A Systematic Benchmarking of Tool-enhanced Emotional Support Dialogue Agentcited by 2
- 2026HUSH-Bench: Measuring Memory-Use Boundaries for Sensitive History in Conversational Agentscited by 2
- 2026Personalization Meets Safety:Mechanisms,Risks,and Mitigations in Personalized LLMscited by 0
- 2026MemLeak: Cross-User Semantic Leakage in Multi-Tenant AI Agent Memorycited by 0
- 2026MemLeak: Cross-User Semantic Leakage in Multi-Tenant AI Agent Memory3 Oct 2026
- 2026Personalization Meets Safety:Mechanisms,Risks,and Mitigations in Personalized LLMs8 Jun 2026
- 2026HUSH-Bench: Measuring Memory-Use Boundaries for Sensitive History in Conversational Agents4 Jun 2026
- 2026Beyond Similarity: Trustworthy Memory Search for Personal AI Agents4 Jun 2026
- 2026A Survey on Long-Term Memory Security in LLM Agents: Attacks, Defenses, and Governance Across the Memory Lifecycle17 Apr 2026
- 2026BenchPreS: A Benchmark for Context-Aware Personalized Preference Selectivity of Persistent-Memory LLMs17 Mar 2026
- 2026TEA-Bench: A Systematic Benchmarking of Tool-enhanced Emotional Support Dialogue Agent26 Jan 2026
Similar, not cited
Close in wording, no citation either way
- 2025Personalized Safety in LLMs: A Benchmark and A Planning-Based Agent Approach
- 2026When Routine Chats Turn Toxic: Unintended Long-Term State Poisoning in Personalized Agents
- 2024Exploring Safety-Utility Trade-Offs in Personalized Language Models
- 2026MemEvoBench: Benchmarking Safety Risks from Memory Misevolution in LLM Agents
- 2026PERMA: Benchmarking Personalized Memory Agents via Event-Driven Preference and Realistic Task Environments
- 2025Enabling Personalized Long-term Interactions in LLM-based Agents through Persistent Memory and User Profiles
- 2026Setoka: A Benchmark for Hierarchical User Understanding in Personalized Agents over Heterogeneous Data
- 2026APeB: Benchmarking Personalization Ability of Large Language Model Agents