Data to 14 Oct 2026

Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Models

arXiv 2305.15594 ↗First posted 24 May 2023cs.LG cs.CL cs.CRcited by 83

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) are excellent in-context learners. However, the sensitivity of data contained in prompts raises privacy concerns. Our work first shows that these concerns are valid: we instantiate a simple but highly effective membership inference attack against the data used to prompt LLMs. To address this vulnerability, one could forego prompting and resort to fine-tuning LLMs with known algorithms for private gradient descent. However, this comes at the expense of the practicality and efficiency offered by prompting. Therefore, we propose to privately learn to prompt. We first show that soft prompts can be obtained privately through gradient descent on downstream data. However, this is not the case for discrete prompts. Thus, we orchestrate a noisy vote among an ensemble of LLMs presented with different prompts, i.e., a flock of stochastic parrots. The vote privately transfers the flock's knowledge into a single public prompt. We show that LLMs prompted with our private algorithms closely match the non-private baselines. For example, using GPT3 as the base model, we achieve a downstream accuracy of 92.7% on the sst2 dataset with (ε=0.147, δ=10-6)-differential privacy vs. 95.2% for the non-private baseline. Through our experiments, we also show that our prompt-based approach is easily deployed with existing commercial APIs.

Key facts. Cited by 83 papers in the map (data through 14 Oct 2026). Builds on 37 of its 58 references that are in the map. First posted on arXiv 24 May 2023.

Builds on

37 of 58 references.

References · 37

2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
2020Language Models are Few-Shot Learnerscited by 32,419
2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
2016Deep Learning with Differential Privacycited by 2,728
2015Character-level Convolutional Networks for Text Classificationcited by 2,330
2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
2020Extracting Training Data from Large Language Modelscited by 2,189

25 more

2019Language Models as Knowledge Bases?cited by 1,936
2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
2019How Can We Know What Language Models Know?cited by 959
2021Membership Inference Attacks From First Principlescited by 878
2022Quantifying Memorization Across Neural Language Modelscited by 763
2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 689
2021Large Language Models Can Be Strong Differentially Private Learnerscited by 412
2021Differentially Private Fine-tuning of Language Modelscited by 400
2016Semi-supervised Knowledge Transfer for Deep Learning from Private Training Datacited by 348
2018Scalable Private Learning with PATEcited by 269
2020Differentially Private Learning Needs Better Features (or Much More Data)cited by 235
2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 220
2021How Many Data Points is a Prompt Worth?cited by 198
2019Commonsense Knowledge Mining from Pretrained Modelscited by 189
2021Counterfactual Memorization in Neural Language Modelscited by 156
2021How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVENcited by 124
2017Bilateral Multi-Perspective Matching for Natural Language Sentencescited by 121
2021Large-Scale Differentially Private BERTcited by 118
2020Revisiting Membership Inference Under Realistic Assumptionscited by 112
2022Memorization in NLP Fine-tuning Methodscited by 45
2021Efficient (Soft) Q-Learning for Text Generation with Limited Good Datacited by 37
2021CaPC Learning: Confidential and Private Collaborative Learningcited by 33
2021How BPE Affects Memorization in Transformerscited by 29

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 12; 2024: 29; 2025: 25; 2026: 17.

83 papers.

Sort:

By area:

2023Ring-A-Bell! How Reliable are Concept Removal Methods for Diffusion Models?cited by 250
2023Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theorycited by 230
2024LLM Dataset Inference: Did you train on my dataset?cited by 110
2023Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generationcited by 96
2024Safeguarding Large Language Models: A Surveycited by 78
2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 66
2024Building Guardrails for Large Language Modelscited by 65
2023DP-OPT: Make Large Language Model Your Privacy-Preserving Prompt Engineercited by 57
2023Privacy in Large Language Models: Attacks, Defenses and Future Directionscited by 51
2024Private Fine-tuning of Large Language Models with Zeroth-order Optimizationcited by 41
2024Privacy-Preserving Retrieval-Augmented Generation with Differential Privacycited by 39
2024Privacy-Preserving Instructions for Aligning Large Language Modelscited by 35

38 more

2024AI Safety in Generative AI Large Language Models: A Surveycited by 33
2023Split-and-Denoise: Protect large language model inference with local differential privacycited by 32
2023Identifying and Mitigating Privacy Risks Stemming from Language Models: A Surveycited by 30
2024PrE-Text: Training Language Models on Private Federated Data in the Age of LLMscited by 29
2024Privacy Preserving Prompt Engineering: A Surveycited by 25
2024Mind the Privacy Unit! User-Level Differential Privacy for Language Model Fine-Tuningcited by 24
2024Private prediction for large-scale synthetic text generationcited by 24
2025Privacy Auditing of Large Language Modelscited by 23
2025A Survey on Privacy Risks and Protection in Large Language Modelscited by 21
2023Privately Aligning Language Models with Reinforcement Learningcited by 17
2024Shake to Leak: Fine-tuning Diffusion Models Can Amplify the Generative Privacy Riskcited by 17
2024Ward: Provable RAG Dataset Inference via LLM Watermarkscited by 16
2025Synthesizing Privacy-Preserving Text Data via Finetuning without Finetuning Billion-Scale LLMscited by 16
2025Differentially Private Steering for Large Language Model Alignmentcited by 15
2025Privacy Attacks on Image AutoRegressive Modelscited by 15
2023Last One Standing: A Comparative Analysis of Security and Privacy of Soft Prompt Tuning, LoRA, and In-Context Learningcited by 14
2024Differentially Private Zeroth-Order Methods for Scalable Large Language Model Finetuningcited by 14
2024The Fire Thief Is Also the Keeper: Balancing Usability and Privacy in Promptscited by 13
2024Open LLMs are Necessary for Current Private Adaptations and Outperform their Closed Alternativescited by 12
2024DP-TabICL: In-Context Learning with Differentially Private Tabular Datacited by 11
2025The Stochastic Parrot on LLM's Shoulder: A Summative Assessment of Physical Concept Understandingcited by 11
2024Data-adaptive Differentially Private Prompt Synthesis for In-Context Learningcited by 10
2025SOFT: Selective Data Obfuscation for Protecting LLM Fine-tuning against Membership Inference Attackscited by 10
2025InvisibleInk: High-Utility and Low-Cost Text Generation with Differential Privacycited by 10
2024Casper: Prompt Sanitization for Protecting User Privacy in Web-Based Large Language Modelscited by 8
2024Certification for Differentially Private Prediction in Gradient-Based Trainingcited by 7
2025Unlocking Post-hoc Dataset Inference with Synthetic Datacited by 7
2024Privacy-preserving Fine-tuning of Large Language Models through Flatnesscited by 6
2025Beyond Data Privacy: New Privacy Risks for Large Language Modelscited by 6
2024LLM-Agent-UMF: LLM-based Agent Unified Modeling Framework for Seamless Design of Multi Active/Passive Core-Agent Architecturescited by 5
2025Clustering and Median Aggregation Improve Differentially Private Inferencecited by 5
2023A Unified View of Differentially Private Deep Generative Modelingcited by 4
2024Adaptively Private Next-Token Prediction of Large Language Modelscited by 4
2025Towards Harnessing the Collaborative Power of Large and Small Models for Domain Taskscited by 4
2025ContextLeak: Auditing Leakage in Private In-Context Learning Methodscited by 4
2026Benchmarking Empirical Privacy Protection for Adaptations of Large Language Modelscited by 4
2023Hot PATE: Private Aggregation of Distributions for Diverse Taskcited by 3
2023ConfusionPrompt: Practical Private Inference for Online Large Language Modelscited by 3

50 most cited of 83 All 83 (JSON)

Similar, not cited

Close in wording, no citation either way