# "It's a Fair Game", or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational Agents

arXiv: https://arxiv.org/abs/2309.11653 · HTML: https://mlsanity.com/paper/2309.11653/

## Key facts

- Cited by 60 papers in the map (data through 14 Oct 2026).
- Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use.
- Builds on 25 of its 89 references that are in the map.
- First posted on arXiv 20 Sep 2023.

## Placed in

- LLM agents and tool use / [Agent threats, guardrails and governance](https://mlsanity.com/agents/agent-threats-guardrails-and-governance/)

## Builds on (most cited first, 25 of 25)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Ethical and social risks of harm from Language Models](https://mlsanity.com/paper/2112.04359/) (2021)
- [Quantifying Memorization Across Neural Language Models](https://mlsanity.com/paper/2202.07646/) (2022)
- [Knowledge Unlearning for Mitigating Privacy Risks in Language Models](https://mlsanity.com/paper/2210.01504/) (2022)
- [Large Language Models Can Be Strong Differentially Private Learners](https://mlsanity.com/paper/2110.05679/) (2021)
- [Differentially Private Fine-tuning of Language Models](https://mlsanity.com/paper/2110.06500/) (2021)
- [Multi-step Jailbreaking Privacy Attacks on ChatGPT](https://mlsanity.com/paper/2304.05197/) (2023)
- [Deduplicating Training Data Mitigates Privacy Risks in Language Models](https://mlsanity.com/paper/2202.06539/) (2022)
- [EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought](https://mlsanity.com/paper/2305.15021/) (2023)
- [The False Promise of Imitating Proprietary LLMs](https://mlsanity.com/paper/2305.15717/) (2023)
- [HuatuoGPT, towards Taming Language Model to Be a Doctor](https://mlsanity.com/paper/2305.15075/) (2023)
- [What Does it Mean for a Language Model to Preserve Privacy?](https://mlsanity.com/paper/2202.05520/) (2022)
- [Counterfactual Memorization in Neural Language Models](https://mlsanity.com/paper/2112.12938/) (2021)
- [ProPILE: Probing Privacy Leakage in Large Language Models](https://mlsanity.com/paper/2307.01881/) (2023)
- [AI Transparency in the Age of LLMs: A Human-Centered Research Roadmap](https://mlsanity.com/paper/2306.01941/) (2023)
- [LLM Platform Security: Applying a Systematic Evaluation Framework to OpenAI's ChatGPT Plugins](https://mlsanity.com/paper/2309.10254/) (2023)
- [Differentially Private Decoding in Large Language Models](https://mlsanity.com/paper/2205.13621/) (2022)
- [An Efficient DP-SGD Mechanism for Large Scale NLP Models](https://mlsanity.com/paper/2107.14586/) (2021)
- [Towards Better Instruction Following Language Models for Chinese: Investigating the Impact of Training Data and Evaluation](https://mlsanity.com/paper/2304.07854/) (2023)
- [Improved Trust in Human-Robot Collaboration with ChatGPT](https://mlsanity.com/paper/2304.12529/) (2023)
- [Several categories of Large Language Models (LLMs): A Short Survey](https://mlsanity.com/paper/2307.10188/) (2023)
- [Building chatbots from large scale domain-specific knowledge bases: challenges and opportunities](https://mlsanity.com/paper/2001.00100/) (2019)

## Built on by (most cited first, 30 of 60)

- [Evaluation and Benchmarking of LLM Agents: A Survey](https://mlsanity.com/paper/2507.21504/) (2025)
- [The Emerged Security and Privacy of LLM Agent: A Survey with Case Studies](https://mlsanity.com/paper/2407.19354/) (2024)
- [Plan-Then-Execute: An Empirical Study of User Trust and Team Performance When Using LLM Agents As A Daily Assistant](https://mlsanity.com/paper/2502.01390/) (2025)
- [Privacy-Preserving Instructions for Aligning Large Language Models](https://mlsanity.com/paper/2402.13659/) (2024)
- [Towards interactive evaluations for interaction harms in human-AI systems](https://mlsanity.com/paper/2405.10632/) (2024)
- [DarkBench: Benchmarking Dark Patterns in Large Language Models](https://mlsanity.com/paper/2503.10728/) (2025)
- [Protecting Users From Themselves: Safeguarding Contextual Privacy in Interactions with Conversational Agents](https://mlsanity.com/paper/2502.18509/) (2025)
- [Position: Privacy Is Not Just Memorization!](https://mlsanity.com/paper/2510.01645/) (2025)
- [Rescriber: Smaller-LLM-Powered User-Led Data Minimization for LLM-Based Chatbots](https://mlsanity.com/paper/2410.11876/) (2024)
- [Understanding the LLM-ification of CHI: Unpacking the Impact of LLMs at CHI through a Systematic Literature Review](https://mlsanity.com/paper/2501.12557/) (2025)
- [Embedding Large Language Models into Extended Reality: Opportunities and Challenges for Inclusion, Engagement, and Privacy](https://mlsanity.com/paper/2402.03907/) (2024)
- [Privacy Leakage Overshadowed by Views of AI: A Study on Human Oversight of Privacy in Language Model Agent](https://mlsanity.com/paper/2411.01344/) (2024)
- [Characterizing and modeling harms from interactions with design patterns in AI interfaces](https://mlsanity.com/paper/2404.11370/) (2024)
- [Toward a Human-Centered Evaluation Framework for Trustworthy LLM-Powered GUI Agents](https://mlsanity.com/paper/2504.17934/) (2025)
- [Invisible Prompts, Visible Threats: Malicious Font Injection in External Resources for Large Language Models](https://mlsanity.com/paper/2505.16957/) (2025)
- [Human-Centered Privacy Research in the Age of Large Language Models](https://mlsanity.com/paper/2402.01994/) (2024)
- [Secret Use of Large Language Model (LLM)](https://mlsanity.com/paper/2409.19450/) (2024)
- [Understanding and Mitigating Bias Inheritance in LLM-based Data Augmentation on Downstream Tasks](https://mlsanity.com/paper/2502.04419/) (2025)
- ['Since Lawyers are Males..': Examining Implicit Gender Bias in Hindi Language Generation by LLMs](https://mlsanity.com/paper/2409.13484/) (2024)
- [Private Yet Social: How LLM Chatbots Support and Challenge Eating Disorder Recovery](https://mlsanity.com/paper/2412.11656/) (2024)
- [Operationalizing Data Minimization for Privacy-Preserving LLM Prompting](https://mlsanity.com/paper/2510.03662/) (2025)
- [Privacy Bias in Language Models: A Contextual Integrity-based Auditing Metric](https://mlsanity.com/paper/2409.03735/) (2024)
- [DAWN: Designing Distributed Agents in a Worldwide Network](https://mlsanity.com/paper/2410.22339/) (2024)
- [The Impact of a Chatbot's Ephemerality-Framing on Self-Disclosure Perceptions](https://mlsanity.com/paper/2505.20464/) (2025)
- [Do Large Language Models Possess Sensitive to Sentiment?](https://mlsanity.com/paper/2409.02370/) (2024)
- [It's My Agent, Not Me: Personal Privacy Boundaries in AI-Delegated Information Sharing](https://mlsanity.com/paper/2509.21712/) (2025)
- [Hear You in Silence: Designing for Active Listening in Human Interaction with Conversational Agents Using Context-Aware Pacing](https://mlsanity.com/paper/2602.06134/) (2026)
- [Learned, Lagged, LLM-splained: LLM Responses to End User Security Questions](https://mlsanity.com/paper/2411.14571/) (2024)
- [Toward Knowledge-Guided AI for Inverse Design in Manufacturing: A Perspective on Domain, Physics, and Human-AI Synergy](https://mlsanity.com/paper/2506.00056/) (2025)
- [ISACL: Internal State Analyzer for Copyrighted Training Data Leakage](https://mlsanity.com/paper/2508.17767/) (2025)

## Abstract (arXiv)

The widespread use of Large Language Model (LLM)-based conversational agents (CAs), especially in high-stakes domains, raises many privacy concerns. Building ethical LLM-based CAs that respect user privacy requires an in-depth understanding of the privacy risks that concern users the most. However, existing research, primarily model-centered, does not provide insight into users' perspectives. To bridge this gap, we analyzed sensitive disclosures in real-world ChatGPT conversations and conducted semi-structured interviews with 19 LLM-based CA users. We found that users are constantly faced with trade-offs between privacy, utility, and convenience when using LLM-based CAs. However, users' erroneous mental models and the dark patterns in system design limited their awareness and comprehension of the privacy risks. Additionally, the human-like interactions encouraged more sensitive disclosures, which complicated users' ability to navigate the trade-offs. We discuss practical design guidelines and the needs for paradigm shifts to protect the privacy of LLM-based CA users.