# It's My Agent, Not Me: Personal Privacy Boundaries in AI-Delegated Information Sharing

arXiv: https://arxiv.org/abs/2509.21712 · HTML: https://mlsanity.com/paper/2509.21712/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 139 references that are in the map.
- First posted on arXiv 26 Sep 2025.

## Builds on (most cited first, 25 of 25)

- [Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory](https://mlsanity.com/paper/2310.17884/) (2023)
- [Are Large Pre-Trained Language Models Leaking Your Personal Information?](https://mlsanity.com/paper/2205.12628/) (2022)
- [What Does it Mean for a Language Model to Preserve Privacy?](https://mlsanity.com/paper/2202.05520/) (2022)
- [A Roadmap to Pluralistic Alignment](https://mlsanity.com/paper/2402.05070/) (2024)
- [PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Action](https://mlsanity.com/paper/2409.00138/) (2024)
- [Eliciting Human Preferences with Language Models](https://mlsanity.com/paper/2310.11589/) (2023)
- [Aligning LLM Agents by Learning Latent Preference from User Edits](https://mlsanity.com/paper/2404.15269/) (2024)
- ["It's a Fair Game", or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational Agents](https://mlsanity.com/paper/2309.11653/) (2023)
- [Reducing Privacy Risks in Online Self-Disclosures with Language Models](https://mlsanity.com/paper/2311.09538/) (2023)
- [Operationalizing Contextual Integrity in Privacy-Conscious Assistants](https://mlsanity.com/paper/2408.02373/) (2024)
- [Aligning Language Models with Demonstrated Feedback](https://mlsanity.com/paper/2406.00888/) (2024)
- [Contextual Integrity in LLMs via Reasoning and Reinforcement Learning](https://mlsanity.com/paper/2506.04245/) (2025)
- [Privacy Checklist: Privacy Violation Detection Grounding on Contextual Integrity Theory](https://mlsanity.com/paper/2408.10053/) (2024)
- [Rescriber: Smaller-LLM-Powered User-Led Data Minimization for LLM-Based Chatbots](https://mlsanity.com/paper/2410.11876/) (2024)
- [Privacy Leakage Overshadowed by Views of AI: A Study on Human Oversight of Privacy in Language Model Agent](https://mlsanity.com/paper/2411.01344/) (2024)
- [Exploring User Security and Privacy Attitudes and Concerns Toward the Use of General-Purpose LLM Chatbots for Mental Health](https://mlsanity.com/paper/2507.10695/) (2025)
- [LLM-as-a-Judge for Privacy Evaluation? Exploring the Alignment of Human and LLM Perceptions of Privacy in Textual Data](https://mlsanity.com/paper/2508.12158/) (2025)
- [Understanding Users' Security and Privacy Concerns and Attitudes Towards Conversational AI Platforms](https://mlsanity.com/paper/2504.06552/) (2025)
- [Scrolling in the Deep: Analysing Contextual Influences on Intervention Effectiveness during Infinite Scrolling on Social Media](https://mlsanity.com/paper/2501.11814/) (2025)
- [Stop the Spread: A Contextual Integrity Perspective on the Appropriateness of COVID-19 Vaccination Certificates](https://mlsanity.com/paper/2205.09036/) (2022)
- [Artificial Intelligence in Deliberation: The AI Penalty and the Emergence of a New Deliberative Divide](https://mlsanity.com/paper/2503.07690/) (2025)
- [Towards Aligning Personalized Conversational Recommendation Agents with Users' Privacy Preferences](https://mlsanity.com/paper/2508.07672/) (2025)
- [A Utility-Theoretic Approach to Privacy in Online Services](https://mlsanity.com/paper/1401.3859/) (2014)
- [User Privacy Harms and Risks in Conversational AI: A Proposed Framework](https://mlsanity.com/paper/2402.09716/) (2024)
- [Personalizing Privacy Protection With Individuals' Regulatory Focus: Would You Preserve or Enhance Your Information Privacy?](https://mlsanity.com/paper/2402.17838/) (2024)

## Built on by (most cited first, 3 of 3)

- [Position: Privacy Is Not Just Memorization!](https://mlsanity.com/paper/2510.01645/) (2025)
- [Personalizing Agent Privacy Decisions via Logical Entailment](https://mlsanity.com/paper/2512.05065/) (2025)
- [From Fragmentation to Integration: Exploring the Design Space of AI Agents for Human-as-the-Unit Privacy Management](https://mlsanity.com/paper/2602.05016/) (2026)

## Abstract (arXiv)

Aligning AI systems with human privacy preferences requires understanding individuals' nuanced disclosure behaviors beyond general privacy norms. Yet eliciting these boundaries remains challenging because they involve context-dependent trade-offs. We present an elicitation approach that probes individuals' privacy boundaries through a discriminative task. Through a between-subjects study with varying communication roles and delegation conditions, we collected 1,650 privacy boundaries from 169 participants across 60 scenarios. We examined how communication roles, AI delegation, and individual differences influence boundaries. Results show that communication roles significantly influence sensitivity to disclosure granularity or identifiability, depending on whose private information is shared; AI delegation and individuals' need for privacy heighten sensitivity to disclosed identifiers. Our analysis further reveals how AI delegation shifts privacy preferences across contexts, highlighting new complexities in AI-mediated communication. We advocate eliciting nuanced privacy boundaries within real-world data flows by considering roles and AI delegation, and enabling user involvement in guiding privacy-aware AI behavior.