mlsanityData to 13 Oct 2026

Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generation

arXiv 2309.11765 ↗First posted 21 Sep 2023cs.LG cs.CRcited by 96

Not on the map. Why?

Abstract (arXiv)

We study the problem of in-context learning (ICL) with large language models (LLMs) on private datasets. This scenario poses privacy risks, as LLMs may leak or regurgitate the private examples demonstrated in the prompt. We propose a novel algorithm that generates synthetic few-shot demonstrations from the private dataset with formal differential privacy (DP) guarantees, and show empirically that it can achieve effective ICL. We conduct extensive experiments on standard benchmarks and compare our algorithm with non-private ICL and zero-shot solutions. Our results demonstrate that our algorithm can achieve competitive performance with strong privacy levels. These results open up new possibilities for ICL with privacy protection for a broad range of applications.

Key facts. Cited by 96 papers in the map (data through 13 Oct 2026). Builds on 37 of its 54 references that are in the map. First posted on arXiv 21 Sep 2023.

Builds on

37 of 54 references.

References · 37

  1. 2020Language Models are Few-Shot Learnerscited by 32,320
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,376
  3. 2019The Curious Case of Neural Text Degenerationcited by 2,893
  4. 2016Deep Learning with Differential Privacycited by 2,718
  5. 2015Character-level Convolutional Networks for Text Classificationcited by 2,319
  6. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,273
  7. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,424
  8. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,376
  9. 2018Hierarchical Neural Story Generationcited by 1,186
  10. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 925
  11. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 515
  12. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 412
25 more
  1. 2021Differentially Private Fine-tuning of Language Modelscited by 398
  2. 2016Semi-supervised Knowledge Transfer for Deep Learning from Private Training Datacited by 347
  3. 2018Scalable Private Learning with PATEcited by 268
  4. 2018Privacy Amplification by Subsampling: Tight Analyses via Couplings and Divergencescited by 240
  5. 2018Improving the Gaussian Mechanism for Differential Privacy: Analytical Calibration and Optimal Denoisingcited by 218
  6. 2021Numerical Composition of Differential Privacycited by 177
  7. 2019Privacy- and Utility-Preserving Textual Analysis via Calibrated Multivariate Perturbationscited by 118
  8. 2020BERT-of-Theseus: Compressing BERT by Progressive Module Replacingcited by 112
  9. 2022Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipecited by 108
  10. 2023Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Modelscited by 82
  11. 2022Differentially Private Language Models for Secure Data Sharingcited by 68
  12. 2022The Limits of Word Level Differential Privacycited by 63
  13. 2022A Customized Text Sanitization Mechanism with Differential Privacycited by 51
  14. 2023Privacy-Preserving In-Context Learning for Large Language Modelscited by 50
  15. 2021TEM: High Utility Metric Differential Privacy on Textcited by 43
  16. 2020A Differentially Private Text Perturbation Method Using a Regularized Mahalanobis Metriccited by 41
  17. 2022Differentially Private Decoding in Large Language Modelscited by 34
  18. 2022Submix: Practical Private Prediction for Large-Scale Language Modelscited by 31
  19. 2022A Critical Review on the Use (and Misuse) of Differential Privacy in Machine Learningcited by 27
  20. 2020Permute-and-Flip: A new mechanism for differentially private selectioncited by 22
  21. 2023Are Chatbots Ready for Privacy-Sensitive Applications? An Investigation into Input Regurgitation and Prompt-Induced Sanitizationcited by 17
  22. 2021The Permute-and-Flip Mechanism is Identical to Report-Noisy-Max with Exponential Noisecited by 14
  23. 2023Guiding Text-to-Text Privatization by Syntaxcited by 14
  24. 2023Driving Context into Text-to-Text Privatizationcited by 13
  25. 2022Privacy-Preserving Domain Adaptation of Semantic Parserscited by 12

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 7; 2024: 24; 2025: 36; 2026: 29.

96 papers.

Sort:

By area:

  1. 2023Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theorycited by 223
  2. 2024ReCaLL: Membership Inference via Relative Conditional Log-Likelihoodscited by 63
  3. 2023DP-OPT: Make Large Language Model Your Privacy-Preserving Prompt Engineercited by 57
  4. 2024Mitigating the Privacy Issues in Retrieval-Augmented Generation (RAG) via Pure Synthetic Datacited by 52
  5. 2024Trust No Bot: Discovering Personal Disclosures in Human-LLM Conversations in the Wildcited by 46
  6. 2024Membership Inference Attacks Against In-Context Learningcited by 42
  7. 2024Private Fine-tuning of Large Language Models with Zeroth-order Optimizationcited by 41
  8. 2024LLM-PBE: Assessing Data Privacy in Large Language Modelscited by 41
  9. 2024Privacy-Preserving Retrieval-Augmented Generation with Differential Privacycited by 39
  10. 2024Privacy-Preserving Instructions for Aligning Large Language Modelscited by 35
  11. 2023Identifying and Mitigating Privacy Risks Stemming from Language Models: A Surveycited by 30
  12. 2024PrE-Text: Training Language Models on Private Federated Data in the Age of LLMscited by 29
38 more
  1. 2023DPZero: Private Fine-Tuning of Language Models without Backpropagationcited by 26
  2. 2025Protecting Users From Themselves: Safeguarding Contextual Privacy in Interactions with Conversational Agentscited by 26
  3. 2024Privacy Preserving Prompt Engineering: A Surveycited by 25
  4. 2024Private prediction for large-scale synthetic text generationcited by 24
  5. 2024Deconstructing The Ethics of Large Language Models from Long-standing Issues to New-emerging Dilemmas: A Surveycited by 23
  6. 2024Mind the Privacy Unit! User-Level Differential Privacy for Language Model Fine-Tuningcited by 23
  7. 2025Privacy Auditing of Large Language Modelscited by 23
  8. 2024Con-ReCall: Detecting Pre-training Data in LLMs via Contrastive Decodingcited by 20
  9. 2025The Canary's Echo: Auditing Privacy Risks of LLM-Generated Synthetic Textcited by 18
  10. 2024RemoteRAG: A Privacy-Preserving LLM Cloud RAG Servicecited by 17
  11. 2025Differentially Private Steering for Large Language Model Alignmentcited by 15
  12. 2025Synthesizing Privacy-Preserving Text Data via Finetuning without Finetuning Billion-Scale LLMscited by 15
  13. 2023Last One Standing: A Comparative Analysis of Security and Privacy of Soft Prompt Tuning, LoRA, and In-Context Learningcited by 14
  14. 2023How Well Do Large Language Models Truly Ground?cited by 14
  15. 2024The Fire Thief Is Also the Keeper: Balancing Usability and Privacy in Promptscited by 13
  16. 2024Open LLMs are Necessary for Current Private Adaptations and Outperform their Closed Alternativescited by 12
  17. 2024DP-TabICL: In-Context Learning with Differentially Private Tabular Datacited by 11
  18. 2025DPImageBench: A Unified Benchmark for Differentially Private Image Synthesiscited by 11
  19. 2024Data-adaptive Differentially Private Prompt Synthesis for In-Context Learningcited by 10
  20. 2025InvisibleInk: High-Utility and Low-Cost Text Generation with Differential Privacycited by 10
  21. 2025Safeguarding Privacy of Retrieval Data against Membership Inference Attacks: Is This Query Too Close to Home?cited by 9
  22. 2025SynthTextEval: Synthetic Text Data Generation and Evaluation for High-Stakes Domainscited by 7
  23. 2025Struct-Bench: A Benchmark for Differentially Private Structured Text Generationcited by 7
  24. 2025Scaling Textual Gradients via Sampling-Based Momentumcited by 6
  25. 2025PPMI: Privacy-Preserving LLM Interaction with Socratic Chain-of-Thought Reasoning and Homomorphically Encrypted Vector Databasescited by 6
  26. 2025Beyond Data Privacy: New Privacy Risks for Large Language Modelscited by 6
  27. 2025Differentially Private Synthetic Text Generation for Retrieval-Augmented Generation (RAG)cited by 6
  28. 2025Clustering and Median Aggregation Improve Differentially Private Inferencecited by 5
  29. 2026PAC-BENCH: Evaluating Multi-Agent Collaboration under Privacy Constraintscited by 5
  30. 2024Adaptively Private Next-Token Prediction of Large Language Modelscited by 4
  31. 2025SynBench: A Benchmark for Differentially Private Text Generationcited by 4
  32. 2025ContextLeak: Auditing Leakage in Private In-Context Learning Methodscited by 4
  33. 2026Observable Channels, Not Just Storage: Evaluating Privacy Leakage in LLM Agent Pipelinescited by 4
  34. 2023Hot PATE: Private Aggregation of Distributions for Diverse Taskcited by 3
  35. 2025Private-RAG: Answering Multiple Queries with LLMs while Keeping Your Data Privatecited by 3
  36. 2026Need to Know: Contextual-Integrity-Grounded Query Rewriting for Privacy-Conscious LLM Delegationcited by 3
  37. 2024Privacy Profiles for Private Selectioncited by 2
  38. 2024AI Delegates with a Dual Focus: Ensuring Privacy and Strategic Self-Disclosurecited by 2

50 most cited of 96 All 96 (JSON)