# Operationalizing Data Minimization for Privacy-Preserving LLM Prompting

arXiv: https://arxiv.org/abs/2510.03662 · HTML: https://mlsanity.com/paper/2510.03662/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 31 references that are in the map.
- First posted on arXiv 4 Oct 2025.

## Builds on (most cited first, 20 of 20)

- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Exams](https://mlsanity.com/paper/2009.13081/) (2020)
- [WildChat: 1M ChatGPT Interaction Logs in the Wild](https://mlsanity.com/paper/2405.01470/) (2024)
- [Proving Test Set Contamination in Black Box Language Models](https://mlsanity.com/paper/2310.17623/) (2023)
- [Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory](https://mlsanity.com/paper/2310.17884/) (2023)
- [When Does Pretraining Help? Assessing Self-Supervised Learning for Law and the CaseHOLD Dataset](https://mlsanity.com/paper/2104.08671/) (2021)
- [Beyond Memorization: Violating Privacy Via Inference with Large Language Models](https://mlsanity.com/paper/2310.07298/) (2023)
- [PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Action](https://mlsanity.com/paper/2409.00138/) (2024)
- ["It's a Fair Game", or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational Agents](https://mlsanity.com/paper/2309.11653/) (2023)
- [PAPILLON: Privacy Preservation from Internet-based and Local Language Model Ensembles](https://mlsanity.com/paper/2410.17127/) (2024)
- [Reducing Privacy Risks in Online Self-Disclosures with Language Models](https://mlsanity.com/paper/2311.09538/) (2023)
- [Trust No Bot: Discovering Personal Disclosures in Human-LLM Conversations in the Wild](https://mlsanity.com/paper/2407.11438/) (2024)
- [To Each (Textual Sequence) Its Own: Improving Memorized-Data Unlearning in Large Language Models](https://mlsanity.com/paper/2405.03097/) (2024)
- [PrivaCI-Bench: Evaluating Privacy with Contextual Integrity and Legal Compliance](https://mlsanity.com/paper/2502.17041/) (2025)
- [Protecting Users From Themselves: Safeguarding Contextual Privacy in Interactions with Conversational Agents](https://mlsanity.com/paper/2502.18509/) (2025)
- [Protecting User Privacy in Remote Conversational Systems: A Privacy-Preserving framework based on text sanitization](https://mlsanity.com/paper/2306.08223/) (2023)
- [Rescriber: Smaller-LLM-Powered User-Led Data Minimization for LLM-Based Chatbots](https://mlsanity.com/paper/2410.11876/) (2024)
- [Prεεmpt: Sanitizing Sensitive Prompts for LLMs](https://mlsanity.com/paper/2504.05147/) (2025)
- [Automated Privacy Information Annotation in Large Language Model Interactions](https://mlsanity.com/paper/2505.20910/) (2025)
- [Beyond PII: How Users Attempt to Estimate and Mitigate Implicit LLM Inference](https://mlsanity.com/paper/2509.12152/) (2025)

## Built on by (most cited first, 8 of 8)

- [Minim: Privacy-Aware Minimal View for Agents via Trusted Local Sanitization](https://mlsanity.com/paper/2606.13949/) (2026)
- [Got a Secret? LLM Agents Can't Keep It: Evaluating Privacy in Multi-Agent Systems](https://mlsanity.com/paper/2605.27766/) (2026)
- [Say Something Else: Rethinking Contextual Privacy as Information Sufficiency](https://mlsanity.com/paper/2604.06409/) (2026)
- [LLM Anonymization Against Agentic Re-Identification](https://mlsanity.com/paper/2605.30848/) (2026)
- [Why Trust Your Agent? Empirical Security Gains from TRiSM-Guided Agentic Workflows in Healthcare](https://mlsanity.com/paper/2606.28666/) (2026)
- [PromptPET: Privacy-Utility Optimized Prompt Obfuscation](https://mlsanity.com/paper/2607.02932/) (2026)
- [Confuse the Model, Control the Flow: Understanding and Mitigating Privacy Leakage from LLM Agents with Information Flow Control](https://mlsanity.com/paper/2609.14003/) (2026)
- [PrivacySkills: How Privacy Guidance Shapes Source Selection in LLM Agents](https://mlsanity.com/paper/2609.35937/) (2026)

## Abstract (arXiv)

The rapid deployment of large language models (LLMs) in consumer applications has led to frequent exchanges of personal information. To obtain useful responses, users often share more than necessary, increasing privacy risks via memorization, context-based personalization, or security breaches. We present a framework to formally define and operationalize data minimization: for a given user prompt and response model, quantifying the least privacy-revealing disclosure that maintains utility, and we propose a priority-queue tree search to locate this optimal point within a privacy-ordered transformation space. We evaluated the framework on four datasets spanning open-ended conversations (ShareGPT, WildChat) and knowledge-intensive tasks with single-ground-truth answers (CaseHold, MedQA), quantifying achievable data minimization with nine LLMs as the response model. Our results demonstrate that larger frontier LLMs can tolerate stronger data minimization while maintaining task quality than smaller open-source models (85.7% redaction for GPT-5 vs. 19.3% for Qwen2.5-0.5B). By comparing with our search-derived benchmarks, we find that LLMs struggle to predict optimal data minimization directly, showing a bias toward abstraction that leads to oversharing. This suggests not just a privacy gap, but a capability gap: models may lack awareness of what information they actually need to solve a task.