# Beyond PII: How Users Attempt to Estimate and Mitigate Implicit LLM Inference

arXiv: https://arxiv.org/abs/2509.12152 · HTML: https://mlsanity.com/paper/2509.12152/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 68 references that are in the map.
- First posted on arXiv 15 Sep 2025.

## Builds on (most cited first, 30 of 35)

- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Ethical and social risks of harm from Language Models](https://mlsanity.com/paper/2112.04359/) (2021)
- [Membership Inference Attacks From First Principles](https://mlsanity.com/paper/2112.03570/) (2021)
- [Ignore Previous Prompt: Attack Techniques For Language Models](https://mlsanity.com/paper/2211.09527/) (2022)
- [Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study](https://mlsanity.com/paper/2305.13860/) (2023)
- [Multi-step Jailbreaking Privacy Attacks on ChatGPT](https://mlsanity.com/paper/2304.05197/) (2023)
- [Analyzing Leakage of Personally Identifiable Information in Language Models](https://mlsanity.com/paper/2302.00539/) (2023)
- [Are Large Pre-Trained Language Models Leaking Your Personal Information?](https://mlsanity.com/paper/2205.12628/) (2022)
- [Security and Privacy Challenges of Large Language Models: A Survey](https://mlsanity.com/paper/2402.00888/) (2024)
- [Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks](https://mlsanity.com/paper/2203.03929/) (2022)
- [Beyond Memorization: Violating Privacy Via Inference with Large Language Models](https://mlsanity.com/paper/2310.07298/) (2023)
- [PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Action](https://mlsanity.com/paper/2409.00138/) (2024)
- [How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVEN](https://mlsanity.com/paper/2111.09509/) (2021)
- [Analyzing Information Leakage of Updates to Natural Language Models](https://mlsanity.com/paper/1912.07942/) (2019)
- ["It's a Fair Game", or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational Agents](https://mlsanity.com/paper/2309.11653/) (2023)
- [Reducing Privacy Risks in Online Self-Disclosures with Language Models](https://mlsanity.com/paper/2311.09538/) (2023)
- [Trust No Bot: Discovering Personal Disclosures in Human-LLM Conversations in the Wild](https://mlsanity.com/paper/2407.11438/) (2024)
- [LLM-PBE: Assessing Data Privacy in Large Language Models](https://mlsanity.com/paper/2408.12787/) (2024)
- [Private Attribute Inference from Images with Vision-Language Models](https://mlsanity.com/paper/2404.10618/) (2024)
- [Believing Anthropomorphism: Examining the Role of Anthropomorphic Cues on Trust in Large Language Models](https://mlsanity.com/paper/2405.06079/) (2024)
- [A Synthetic Dataset for Personal Attribute Inference](https://mlsanity.com/paper/2406.07217/) (2024)
- [CLEAR: Towards Contextual LLM-Empowered Privacy Policy Analysis and Risk Generation for Large Language Model Applications](https://mlsanity.com/paper/2410.13387/) (2024)
- [PII-Compass: Guiding LLM training data extraction prompts towards the target PII via grounding](https://mlsanity.com/paper/2407.02943/) (2024)
- [Canary Extraction in Natural Language Understanding Models](https://mlsanity.com/paper/2203.13920/) (2022)
- [Rescriber: Smaller-LLM-Powered User-Led Data Minimization for LLM-Based Chatbots](https://mlsanity.com/paper/2410.11876/) (2024)
- [Understanding the Impact of Long-Term Memory on Self-Disclosure with Large Language Model-Driven Chatbots for Public Health Intervention](https://mlsanity.com/paper/2402.11353/) (2024)
- [Malicious LLM-Based Conversational AI Makes Users Reveal Personal Information](https://mlsanity.com/paper/2506.11680/) (2025)
- [Exploring User Security and Privacy Attitudes and Concerns Toward the Use of General-Purpose LLM Chatbots for Mental Health](https://mlsanity.com/paper/2507.10695/) (2025)
- [Listening to the Voices: Describing Ethical Caveats of Conversational User Interfaces According to Experts and Frequent Users](https://mlsanity.com/paper/2401.14746/) (2024)

## Built on by (most cited first, 2 of 2)

- [Operationalizing Data Minimization for Privacy-Preserving LLM Prompting](https://mlsanity.com/paper/2510.03662/) (2025)
- [PromptPET: Privacy-Utility Optimized Prompt Obfuscation](https://mlsanity.com/paper/2607.02932/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) such as ChatGPT can infer personal attributes from seemingly innocuous text, raising privacy risks beyond memorized data leakage. While prior work has demonstrated these risks, little is known about how users estimate and respond. We conducted a survey with 240 U.S. participants who judged text snippets for inference risks, reported concern levels, and attempted rewrites to block inference. We compared their rewrites with those generated by ChatGPT and Rescriber, a state-of-the-art sanitization tool. Results show that participants struggled to anticipate inference, performing a little better than chance. User rewrites were effective in just 28% of cases - better than Rescriber but worse than ChatGPT. We examined our participants' rewriting strategies, and observed that while paraphrasing was the most common strategy it is also the least effective; instead abstraction and adding ambiguity were more successful. Our work highlights the importance of inference-aware design in LLM interactions.