# Autonomy Reshapes How Personalization Affects Privacy Concerns and Trust in LLM Agents

arXiv: https://arxiv.org/abs/2510.04465 · HTML: https://mlsanity.com/paper/2510.04465/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 57 references that are in the map.
- First posted on arXiv 6 Oct 2025.

## Builds on (most cited first, 20 of 20)

- [Artificial Intelligence, Values and Alignment](https://mlsanity.com/paper/2001.09768/) (2020)
- [Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agents](https://mlsanity.com/paper/2306.03314/) (2023)
- [Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory](https://mlsanity.com/paper/2310.17884/) (2023)
- [Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalization](https://mlsanity.com/paper/2406.01171/) (2024)
- [PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Action](https://mlsanity.com/paper/2409.00138/) (2024)
- [Red-Teaming LLM Multi-Agent Systems via Communication Attacks](https://mlsanity.com/paper/2502.14847/) (2025)
- [Personalization of Large Language Models: A Survey](https://mlsanity.com/paper/2411.00027/) (2024)
- [AirGapAgent: Protecting Privacy-Conscious Conversational Agents](https://mlsanity.com/paper/2405.05175/) (2024)
- [AgentDAM: Privacy Leakage Evaluation for Autonomous Web Agents](https://mlsanity.com/paper/2503.09780/) (2025)
- ["It's a Fair Game", or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational Agents](https://mlsanity.com/paper/2309.11653/) (2023)
- [Levels of Autonomy for AI Agents](https://mlsanity.com/paper/2506.12469/) (2025)
- [Creating General User Models from Computer Use](https://mlsanity.com/paper/2505.10831/) (2025)
- [Characterizing AI Agents for Alignment and Governance](https://mlsanity.com/paper/2504.21848/) (2025)
- [Designing for Human-Agent Alignment: Understanding what humans want from their agents](https://mlsanity.com/paper/2404.04289/) (2024)
- [Privately Aligning Language Models with Reinforcement Learning](https://mlsanity.com/paper/2310.16960/) (2023)
- [How does HCI Understand Human Autonomy and Agency?](https://mlsanity.com/paper/2301.12490/) (2023)
- [Privacy Leakage Overshadowed by Views of AI: A Study on Human Oversight of Privacy in Language Model Agent](https://mlsanity.com/paper/2411.01344/) (2024)
- [LeakAgent: RL-based Red-teaming Agent for LLM Privacy Leakage](https://mlsanity.com/paper/2412.05734/) (2024)
- [When Personalization Meets Reality: A Multi-Faceted Analysis of Personalized Preference Learning](https://mlsanity.com/paper/2502.19158/) (2025)
- [Automation Bias in the AI Act: On the Legal Implications of Attempting to De-Bias Human Oversight of AI](https://mlsanity.com/paper/2502.10036/) (2025)

## Built on by (most cited first, 1 of 1)

- [Reflections and New Directions for Human-Centered Large Language Models](https://mlsanity.com/paper/2605.06901/) (2026)

## Abstract (arXiv)

LLM agents require personal information for personalization in order to effectively act on users' behalf, but this raises privacy concerns that can discourage data sharing, limiting both the autonomy levels at which agents can operate and the effectiveness of personalization. Yet the expanded design space of agent autonomy also presents opportunities to shape these effects, which remain underexplored. We conducted a 3×3 between-subjects experiment (N=450) to study how agent autonomy level influences personalization's effects on users' privacy concerns, trust, and willingness to use, as well as the underlying psychological processes. We find that risk-contingent autonomy, where the agent delegates control back to users upon detecting potential privacy leakage, improves users' perceived control. This in turn attenuates personalization's adverse effects: privacy concerns rise less and trust declines less. Our results suggest that designing agent's autonomy that supports human autonomy (both perceived control and oversight effectiveness) helps users benefit from personalization without being deterred by growing privacy concerns. This positions aligning decision authority between users and agents (autonomy alignment) as an important opportunity and challenge for trustworthy human-agent interaction.