# Locally Differentially Private Document Generation Using Zero Shot Prompting

arXiv: https://arxiv.org/abs/2310.16111 · HTML: https://mlsanity.com/paper/2310.16111/

## Key facts

- Cited by 63 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 67 references that are in the map.
- First posted on arXiv 24 Oct 2023.

## Builds on (most cited first, 30 of 37)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks](https://mlsanity.com/paper/1802.08232/) (2018)
- [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://mlsanity.com/paper/2204.06745/) (2022)
- [Is BERT Really Robust? A Strong Baseline for Natural Language Attack on Text Classification and Entailment](https://mlsanity.com/paper/1907.11932/) (2019)
- [Textbooks Are All You Need II: phi-1.5 technical report](https://mlsanity.com/paper/2309.05463/) (2023)
- [Large Language Models Can Be Strong Differentially Private Learners](https://mlsanity.com/paper/2110.05679/) (2021)
- [PAWS: Paraphrase Adversaries from Word Scrambling](https://mlsanity.com/paper/1904.01130/) (2019)
- [Differentially Private Fine-tuning of Language Models](https://mlsanity.com/paper/2110.06500/) (2021)
- [RAPPOR: Randomized Aggregatable Privacy-Preserving Ordinal Response](https://mlsanity.com/paper/1407.6981/) (2014)
- [Privacy- and Utility-Preserving Textual Analysis via Calibrated Multivariate Perturbations](https://mlsanity.com/paper/1910.08902/) (2019)
- [Differentially Private Language Models for Secure Data Sharing](https://mlsanity.com/paper/2210.13918/) (2022)
- [TAG: Gradient Attack on Transformer-based Language Models](https://mlsanity.com/paper/2103.06819/) (2021)
- [The Limits of Word Level Differential Privacy](https://mlsanity.com/paper/2205.02130/) (2022)
- [LAMP: Extracting Text from Gradients with Language Model Priors](https://mlsanity.com/paper/2202.08827/) (2022)
- [A Customized Text Sanitization Mechanism with Differential Privacy](https://mlsanity.com/paper/2207.01193/) (2022)
- [Differentially Private Language Models Benefit from Public Pre-training](https://mlsanity.com/paper/2009.05886/) (2020)

## Built on by (most cited first, 30 of 63)

- [The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG)](https://mlsanity.com/paper/2402.16893/) (2024)
- [Differentially Private Synthetic Data via Foundation Model APIs 2: Text](https://mlsanity.com/paper/2403.01749/) (2024)
- [DP-OPT: Make Large Language Model Your Privacy-Preserving Prompt Engineer](https://mlsanity.com/paper/2312.03724/) (2023)
- [PrE-Text: Training Language Models on Private Federated Data in the Age of LLMs](https://mlsanity.com/paper/2406.02958/) (2024)
- [PrivacyRestore: Privacy-Preserving Inference in Large Language Models via Privacy Removal and Restoration](https://mlsanity.com/paper/2406.01394/) (2024)
- [Privacy Preserving Prompt Engineering: A Survey](https://mlsanity.com/paper/2404.06001/) (2024)
- [DP-MLM: Differentially Private Text Rewriting Using Masked Language Models](https://mlsanity.com/paper/2407.00637/) (2024)
- [Rethinking Data Protection in the (Generative) Artificial Intelligence Era](https://mlsanity.com/paper/2507.03034/) (2025)
- [Reconstruct Your Previous Conversations! Comprehensively Investigating Privacy Leakage Risks in Conversations with GPT Models](https://mlsanity.com/paper/2402.02987/) (2024)
- [Synthesizing Privacy-Preserving Text Data via Finetuning without Finetuning Billion-Scale LLMs](https://mlsanity.com/paper/2503.12347/) (2025)
- [Can Language Models Follow Multiple Turns of Entangled Instructions?](https://mlsanity.com/paper/2503.13222/) (2025)
- [POPri: Private Federated Learning using Preference-Optimized Synthetic Data](https://mlsanity.com/paper/2504.16438/) (2025)
- [The Fire Thief Is Also the Keeper: Balancing Usability and Privacy in Prompts](https://mlsanity.com/paper/2406.14318/) (2024)
- [1-Diffractor: Efficient and Utility-Preserving Text Obfuscation Leveraging Word-Level Metric Differential Privacy](https://mlsanity.com/paper/2405.01678/) (2024)
- [Data-adaptive Differentially Private Prompt Synthesis for In-Context Learning](https://mlsanity.com/paper/2410.12085/) (2024)
- [InvisibleInk: High-Utility and Low-Cost Text Generation with Differential Privacy](https://mlsanity.com/paper/2507.02974/) (2025)
- [LLM-as-a-Judge for Privacy Evaluation? Exploring the Alignment of Human and LLM Perceptions of Privacy in Textual Data](https://mlsanity.com/paper/2508.12158/) (2025)
- [Just Rewrite It Again: A Post-Processing Method for Enhanced Semantic Similarity and Privacy Preservation of Differentially Private Rewritten Text](https://mlsanity.com/paper/2405.19831/) (2024)
- [NAP^2: A Benchmark for Naturalness and Privacy-Preserving Text Rewriting by Learning from Human](https://mlsanity.com/paper/2406.03749/) (2024)
- [On the Impact of Noise in Differentially Private Text Rewriting](https://mlsanity.com/paper/2501.19022/) (2025)
- [Estimating Privacy Leakage of Augmented Contextual Knowledge in Language Models](https://mlsanity.com/paper/2410.03026/) (2024)
- [DP-Fusion: Token-Level Differentially Private Inference for Large Language Models](https://mlsanity.com/paper/2507.04531/) (2025)
- [Delving into Differentially Private Transformer](https://mlsanity.com/paper/2405.18194/) (2024)
- [TAROT: Task-Oriented Authorship Obfuscation Using Policy Optimization Methods](https://mlsanity.com/paper/2407.21630/) (2024)
- [Legal Fact Prediction: The Missing Piece in Legal Judgment Prediction](https://mlsanity.com/paper/2409.07055/) (2024)
- [Spend Your Budget Wisely: Towards an Intelligent Distribution of the Privacy Budget in Differentially Private Text Rewriting](https://mlsanity.com/paper/2503.22379/) (2025)
- [Reconstruction of Differentially Private Text Sanitization via Large Language Models](https://mlsanity.com/paper/2410.12443/) (2024)
- [ConfusionPrompt: Practical Private Inference for Online Large Language Models](https://mlsanity.com/paper/2401.00870/) (2023)
- [CodeCipher: Learning to Obfuscate Source Code Against LLMs](https://mlsanity.com/paper/2410.05797/) (2024)
- [Leveraging Semantic Triples for Private Document Generation with Local Differential Privacy Guarantees](https://mlsanity.com/paper/2508.20736/) (2025)

## Abstract (arXiv)

Numerous studies have highlighted the privacy risks associated with pretrained large language models. In contrast, our research offers a unique perspective by demonstrating that pretrained large language models can effectively contribute to privacy preservation. We propose a locally differentially private mechanism called DP-Prompt, which leverages the power of pretrained large language models and zero-shot prompting to counter author de-anonymization attacks while minimizing the impact on downstream utility. When DP-Prompt is used with a powerful language model like ChatGPT (gpt-3.5), we observe a notable reduction in the success rate of de-anonymization attacks, showing that it surpasses existing approaches by a considerable margin despite its simpler design. For instance, in the case of the IMDB dataset, DP-Prompt (with ChatGPT) perfectly recovers the clean sentiment F1 score while achieving a 46% reduction in author identification F1 score against static attackers and a 26% reduction against adaptive attackers. We conduct extensive experiments across six open-source large language models, ranging up to 7 billion parameters, to analyze various effects of the privacy-utility tradeoff.