# Trust No Bot: Discovering Personal Disclosures in Human-LLM Conversations in the Wild

arXiv: https://arxiv.org/abs/2407.11438 · HTML: https://mlsanity.com/paper/2407.11438/

## Key facts

- Cited by 47 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 76 references that are in the map.
- First posted on arXiv 16 Jul 2024.

## Builds on (most cited first, 22 of 22)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Tree of Attacks: Jailbreaking Black-Box LLMs Automatically](https://mlsanity.com/paper/2312.02119/) (2023)
- [Scalable Extraction of Training Data from (Production) Language Models](https://mlsanity.com/paper/2311.17035/) (2023)
- [Large Language Models Can Be Strong Differentially Private Learners](https://mlsanity.com/paper/2110.05679/) (2021)
- [Differentially Private Fine-tuning of Language Models](https://mlsanity.com/paper/2110.06500/) (2021)
- [Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning](https://mlsanity.com/paper/2011.01403/) (2020)
- [Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory](https://mlsanity.com/paper/2310.17884/) (2023)
- [Efficient Large Language Models: A Survey](https://mlsanity.com/paper/2312.03863/) (2023)
- [Beyond Memorization: Violating Privacy Via Inference with Large Language Models](https://mlsanity.com/paper/2310.07298/) (2023)
- [A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Models](https://mlsanity.com/paper/2201.05337/) (2022)
- [ProPILE: Probing Privacy Leakage in Large Language Models](https://mlsanity.com/paper/2307.01881/) (2023)
- [Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generation](https://mlsanity.com/paper/2309.11765/) (2023)
- [Opening up ChatGPT: Tracking openness, transparency, and accountability in instruction-tuned text generators](https://mlsanity.com/paper/2307.05532/) (2023)
- [Reducing Privacy Risks in Online Self-Disclosures with Language Models](https://mlsanity.com/paper/2311.09538/) (2023)
- [UX Research on Conversational Human-AI Interaction: A Literature Review of the ACM Digital Library](https://mlsanity.com/paper/2202.09895/) (2022)
- [Position: Key Claims in LLM Research Have a Long Tail of Footnotes](https://mlsanity.com/paper/2308.07120/) (2023)
- [Are Chatbots Ready for Privacy-Sensitive Applications? An Investigation into Input Regurgitation and Prompt-Induced Sanitization](https://mlsanity.com/paper/2305.15008/) (2023)
- [Understanding the Impact of Long-Term Memory on Self-Disclosure with Large Language Model-Driven Chatbots for Public Health Intervention](https://mlsanity.com/paper/2402.11353/) (2024)
- [NLP for Maternal Healthcare: Perspectives and Guiding Principles in the Age of LLMs](https://mlsanity.com/paper/2312.11803/) (2023)
- [Privacy-Preserving Domain Adaptation of Semantic Parsers](https://mlsanity.com/paper/2212.10520/) (2022)
- [Are Neural Topic Models Broken?](https://mlsanity.com/paper/2210.16162/) (2022)
- [Identifying Self-Disclosures of Use, Misuse and Addiction in Community-based Social Media Posts](https://mlsanity.com/paper/2311.09066/) (2023)

## Built on by (most cited first, 30 of 47)

- [Clio: Privacy-Preserving Insights into Real-World AI Use](https://mlsanity.com/paper/2412.13678/) (2024)
- [Imprompter: Tricking LLM Agents into Improper Tool Use](https://mlsanity.com/paper/2410.14923/) (2024)
- [The Personality Illusion: Revealing Dissociation Between Self-Reports & Behavior in LLMs](https://mlsanity.com/paper/2509.03730/) (2025)
- [Protecting Users From Themselves: Safeguarding Contextual Privacy in Interactions with Conversational Agents](https://mlsanity.com/paper/2502.18509/) (2025)
- [Human Decision-making is Susceptible to AI-driven Manipulation](https://mlsanity.com/paper/2502.07663/) (2025)
- [Position: Privacy Is Not Just Memorization!](https://mlsanity.com/paper/2510.01645/) (2025)
- [Rescriber: Smaller-LLM-Powered User-Led Data Minimization for LLM-Based Chatbots](https://mlsanity.com/paper/2410.11876/) (2024)
- [A False Sense of Privacy: Evaluating Textual Data Sanitization Beyond Surface-level Privacy Leakage](https://mlsanity.com/paper/2504.21035/) (2025)
- [Personalized Attacks of Social Engineering in Multi-turn Conversations: LLM Agents for Simulation and Detection](https://mlsanity.com/paper/2503.15552/) (2025)
- [Malicious LLM-Based Conversational AI Makes Users Reveal Personal Information](https://mlsanity.com/paper/2506.11680/) (2025)
- [Humanizing Machines: Rethinking LLM Anthropomorphism Through a Multi-Level Framework of Design](https://mlsanity.com/paper/2508.17573/) (2025)
- [User Privacy and Large Language Models: An Analysis of Frontier Developers' Privacy Policies](https://mlsanity.com/paper/2509.05382/) (2025)
- [Operationalizing Data Minimization for Privacy-Preserving LLM Prompting](https://mlsanity.com/paper/2510.03662/) (2025)
- [Colombian Waitresses y Jueces canadienses: Gender and Country Biases in Occupation Recommendations from LLMs](https://mlsanity.com/paper/2505.02456/) (2025)
- [VortexPIA: Indirect Prompt Injection Attack against LLMs for Efficient Extraction of User Privacy](https://mlsanity.com/paper/2510.04261/) (2025)
- [VoxPrivacy: A Benchmark for Evaluating Interactional Privacy of Speech Language Models](https://mlsanity.com/paper/2601.19956/) (2026)
- [Privacy Bias in Language Models: A Contextual Integrity-based Auditing Metric](https://mlsanity.com/paper/2409.03735/) (2024)
- [Automated Privacy Information Annotation in Large Language Model Interactions](https://mlsanity.com/paper/2505.20910/) (2025)
- [Privacy-R1: Privacy-Aware Multi-LLM Agent Collaboration via Reinforcement Learning](https://mlsanity.com/paper/2510.16054/) (2025)
- [Can Large Language Models Really Recognize Your Name?](https://mlsanity.com/paper/2505.14549/) (2025)
- [Show or Tell? Modeling the evolution of request-making in Human-LLM conversations](https://mlsanity.com/paper/2508.01213/) (2025)
- [The Subtle Art of Defection: Understanding Uncooperative Behaviors in LLM based Multi-Agent Systems](https://mlsanity.com/paper/2511.15862/) (2025)
- [PAAC: Privacy-Aware Agentic Device-Cloud Collaboration](https://mlsanity.com/paper/2605.08646/) (2026)
- [AI Delegates with a Dual Focus: Ensuring Privacy and Strategic Self-Disclosure](https://mlsanity.com/paper/2409.17642/) (2024)
- [Creation of AI-driven Smart Spaces for Enhanced Indoor Environments -- A Survey](https://mlsanity.com/paper/2412.14708/) (2024)
- [Beyond PII: How Users Attempt to Estimate and Mitigate Implicit LLM Inference](https://mlsanity.com/paper/2509.12152/) (2025)
- [PATCH: Mitigating PII Leakage in Language Models with Privacy-Aware Targeted Circuit PatcHing](https://mlsanity.com/paper/2510.07452/) (2025)
- [Characterizing Narrative Content in Web-scale LLM Pretraining Data](https://mlsanity.com/paper/2606.19468/) (2026)
- [AI Fiction in the Wild](https://mlsanity.com/paper/2606.22748/) (2026)
- [User Perceptions vs. Proxy LLM Judges: Privacy and Helpfulness in LLM Responses to Privacy-Sensitive Scenarios](https://mlsanity.com/paper/2510.20721/) (2025)

## Abstract (arXiv)

Measuring personal disclosures made in human-chatbot interactions can provide a better understanding of users' AI literacy and facilitate privacy research for large language models (LLMs). We run an extensive, fine-grained analysis on the personal disclosures made by real users to commercial GPT models, investigating the leakage of personally identifiable and sensitive information. To understand the contexts in which users disclose to chatbots, we develop a taxonomy of tasks and sensitive topics, based on qualitative and quantitative analysis of naturally occurring conversations. We discuss these potential privacy harms and observe that: (1) personally identifiable information (PII) appears in unexpected contexts such as in translation or code editing (48% and 16% of the time, respectively) and (2) PII detection alone is insufficient to capture the sensitive topics that are common in human-chatbot interactions, such as detailed sexual preferences or specific drug use habits. We believe that these high disclosure rates are of significant importance for researchers and data curators, and we call for the design of appropriate nudging mechanisms to help users moderate their interactions.