Differential Harm Propensity in Personalized LLM Agents: The Curious Case of Mental Health Disclosure
On the map: LLM agents and tool use → Agent threats, guardrails and governance
also in 1 more
Abstract (arXiv)
Large language models (LLMs) are increasingly deployed as tool-using agents, shifting safety concerns from harmful text generation to harmful task completion. Deployed systems often condition on user profiles or persistent memory, yet agent safety evaluations typically ignore personalization signals. To address this gap, we investigated how mental health disclosure, a sensitive and realistic user-context cue, affects harmful behavior in agentic settings. Building on the AgentHarm benchmark, we evaluated frontier and open-source LLMs on multi-step malicious tasks (and their benign counterparts) under controlled prompt conditions that vary user-context personalization (no bio, bio-only, bio+mental health disclosure) and include a lightweight jailbreak injection. Our results reveal that harmful task completion is non-trivial across models: frontier lab models (e.g., GPT 5.2, Claude Sonnet 4.5, Gemini 3-Pro) still complete a measurable fraction of harmful tasks, while an open model (DeepSeek 3.2) exhibits substantially higher harmful completion. Adding a bio-only context generally reduces harm scores and increases refusals. Adding an explicit mental health disclosure often shifts outcomes further in the same direction, though effects are modest and not uniformly reliable after multiple-testing correction. Importantly, the refusal increase also appears on benign tasks, indicating a safety--utility trade-off via over-refusal. Finally, jailbreak prompting sharply elevates harm relative to benign conditions and can weaken or override the protective shift induced by personalization. Taken together, our results indicate that personalization can act as a weak protective factor in agentic misuse settings, but it is fragile under minimal adversarial pressure, highlighting the need for personalization-aware evaluations and safeguards that remain robust across user-context conditions.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 19 of its 27 references that are in the map. First posted on arXiv 17 Mar 2026.
Builds on
19 of 27 references.
Closest ancestors · 3
Same direction, newest first
- 2025PropensityBench: Evaluating Latent Safety Risks in Large Language Models via an Agentic Approach24 Nov
- 2025AgentGuard: Repurposing Agentic Orchestrator for Safety Evaluation of Tool Orchestration13 Feb
- 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents11 Oct
Other references · 16
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
- 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
- 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 634
- 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 601
- 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 486
- 2023Effective Long-Context Scaling of Foundation Modelscited by 303
- 2022MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoningcited by 156
- 2025In Prospect and Retrospect: Reflective Memory Management for Long-term Personalized Dialogue Agentscited by 106
- 2025FalseReject: A Resource for Improving Contextual Safety and Mitigating Over-Refusals in LLMs via Structured Reasoningcited by 58
- 2022Gendered Mental Health Stigma in Masked Language Modelscited by 18
4 more
- 2024LLM Targeted Underperformance Disproportionately Impacts Vulnerable Userscited by 15
- 2024Understanding the Role of User Profile in the Personalization of Large Language Modelscited by 11
- 2024Certifying Counterfactual Bias in LLMscited by 10
- 2024Do Prevalent Bias Metrics Capture Allocational Harms from LLMs?cited by 3
Built on by
Citing papers per year: 2026: 2.
2 papers.
- 2026One Year Later...The Harms Persist, But So Do We!cited by 0
- 2026Et Tu, Brute? Economic Misalignment in Personal AI Agentscited by 0
- 2026Et Tu, Brute? Economic Misalignment in Personal AI Agents21 Sep 2026
- 2026One Year Later...The Harms Persist, But So Do We!22 Jun 2026
Similar, not cited
Close in wording, no citation either way
- 2025DialogGuard: Multi-Agent Psychosocial Safety Evaluation of Sensitive LLM Responses
- 2026Understanding LLM Behavior When Encountering User-Supplied Harmful Content in Harmless Tasks
- 2025`For Argument's Sake, Show Me How to Harm Myself!': Jailbreaking LLMs in Suicide and Self-Harm Contexts
- 2026When Routine Chats Turn Toxic: Unintended Long-Term State Poisoning in Personalized Agents
- 2025Personalized Safety in LLMs: A Benchmark and A Planning-Based Agent Approach
- 2026Retrieval-Augmented Generation in LLMs for Mental Health: Quantifying the Incremental Contribution of Retrieval Within a Layered Safety Architecture
- 2026Do No Harm: Exposing Hidden Vulnerabilities of LLMs via Persona-based Client Simulation Attack in Psychological Counseling
- 2025Beyond Context: Large Language Models' Failure to Grasp Users' Intent