mlsanityData to 7 Oct 2026

Autonomy Reshapes How Personalization Affects Privacy Concerns and Trust in LLM Agents

arXiv 2510.04465 ↗First posted 6 Oct 2025cs.HC cs.AI cs.CRcited by 0

Not on the map. Why?

Abstract (arXiv)

LLM agents require personal information for personalization in order to effectively act on users' behalf, but this raises privacy concerns that can discourage data sharing, limiting both the autonomy levels at which agents can operate and the effectiveness of personalization. Yet the expanded design space of agent autonomy also presents opportunities to shape these effects, which remain underexplored. We conducted a 3×3 between-subjects experiment (N=450) to study how agent autonomy level influences personalization's effects on users' privacy concerns, trust, and willingness to use, as well as the underlying psychological processes. We find that risk-contingent autonomy, where the agent delegates control back to users upon detecting potential privacy leakage, improves users' perceived control. This in turn attenuates personalization's adverse effects: privacy concerns rise less and trust declines less. Our results suggest that designing agent's autonomy that supports human autonomy (both perceived control and oversight effectiveness) helps users benefit from personalization without being deterred by growing privacy concerns. This positions aligning decision authority between users and agents (autonomy alignment) as an important opportunity and challenge for trustworthy human-agent interaction.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 20 of its 57 references that are in the map. First posted on arXiv 6 Oct 2025.

Builds on

20 of 57 references.

References · 20

  1. 2020Artificial Intelligence, Values and Alignmentcited by 443
  2. 2023Multi-Agent Collaboration: Harnessing the Power of Intelligent LLM Agentscited by 329
  3. 2023Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theorycited by 218
  4. 2024Two Tales of Persona in LLMs: A Survey of Role-Playing and Personalizationcited by 203
  5. 2024PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Actioncited by 161
  6. 2025Red-Teaming LLM Multi-Agent Systems via Communication Attackscited by 118
  7. 2024Personalization of Large Language Models: A Surveycited by 100
  8. 2024AirGapAgent: Protecting Privacy-Conscious Conversational Agentscited by 90
  9. 2025AgentDAM: Privacy Leakage Evaluation for Autonomous Web Agentscited by 79
  10. 2023"It's a Fair Game", or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational Agentscited by 60
  11. 2025Levels of Autonomy for AI Agentscited by 45
  12. 2025Creating General User Models from Computer Usecited by 44
8 more
  1. 2025Characterizing AI Agents for Alignment and Governancecited by 30
  2. 2024Designing for Human-Agent Alignment: Understanding what humans want from their agentscited by 18
  3. 2023Privately Aligning Language Models with Reinforcement Learningcited by 17
  4. 2023How does HCI Understand Human Autonomy and Agency?cited by 14
  5. 2024Privacy Leakage Overshadowed by Views of AI: A Study on Human Oversight of Privacy in Language Model Agentcited by 14
  6. 2024LeakAgent: RL-based Red-teaming Agent for LLM Privacy Leakagecited by 13
  7. 2025When Personalization Meets Reality: A Multi-Faceted Analysis of Personalized Preference Learningcited by 9
  8. 2025Automation Bias in the AI Act: On the Legal Implications of Attempting to De-Bias Human Oversight of AIcited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026PrivacySkills: How Privacy Guidance Shapes Source Selection in LLM Agents
  2. 2026SovereignPA-Bench: Evaluating User-Owned Personal Agents under Evolving Intent, Platform Mediation, and Consent Constraints
  3. 2026From Fragmentation to Integration: Exploring the Design Space of AI Agents for Human-as-the-Unit Privacy Management
  4. 2026PrivacyAlign: Contextual Privacy Alignment for LLM Agents
  5. 2025The Basic B*** Effect: The Use of LLM-based Agents Reduces the Distinctiveness and Diversity of People's Choices
  6. 2026Assistant or Actor? Student Trust, Control, and Delegation Regret When Using a General-Purpose AI Agent
  7. 2026How Well Can LLM Agents Simulate End-User Security and Privacy Attitudes and Behaviors?
  8. 2026Personalized Privacy Control in LLMs via Attention Head Intervention