mlsanityData to 14 Oct 2026

On the Privacy Risk of In-context Learning

arXiv 2411.10512 ↗First posted 15 Nov 2024cs.LG cs.CRcited by 32

On the map: LLM safety and security → Membership inference attacks

Abstract (arXiv)

Large language models (LLMs) are excellent few-shot learners. They can perform a wide variety of tasks purely based on natural language prompts provided to them. These prompts contain data of a specific downstream task -- often the private dataset of a party, e.g., a company that wants to leverage the LLM for their purposes. We show that deploying prompted models presents a significant privacy risk for the data used within the prompt by instantiating a highly effective membership inference attack. We also observe that the privacy risk of prompted models exceeds fine-tuned models at the same utility levels. After identifying the model's sensitivity to their prompts -- in the form of a significantly higher prediction confidence on the prompted data -- as a cause for the increased risk, we propose ensembling as a mitigation strategy. By aggregating over multiple different versions of a prompted model, membership inference risk can be decreased.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Placed in Membership inference attacks, one of 12 directions in LLM safety and security. Builds on 36 of its 48 references that are in the map. First posted on arXiv 15 Nov 2024.

Builds on

36 of 48 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022RelaxLoss: Defending Membership Inference Attacks without Losing Utility12 Jul
  2. 2021Membership Inference Attacks From First Principles7 Dec
  3. 2021Mitigating Membership Inference Attacks by Self-Distillation Through a Novel Ensemble Architecture15 Oct

Other references · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  4. 2016Deep Learning with Differential Privacycited by 2,728
  5. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  6. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  7. 2020Extracting Training Data from Large Language Modelscited by 2,189
  8. 2019Language Models as Knowledge Bases?cited by 1,936
  9. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  10. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  11. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,195
  12. 2019When Does Label Smoothing Help?cited by 988
21 more
  1. 2019How Can We Know What Language Models Know?cited by 959
  2. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
  3. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  4. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 689
  5. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 412
  6. 2021Differentially Private Fine-tuning of Language Modelscited by 400
  7. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 384
  8. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 377
  9. 2021Factual Probing Is [MASK]: Learning vs. Learning to Recallcited by 291
  10. 2020Revisiting Few-sample BERT Fine-tuningcited by 238
  11. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 220
  12. 2021How Many Data Points is a Prompt Worth?cited by 198
  13. 2019MemGuard: Defending against Black-Box Membership Inference Attacks via Adversarial Examplescited by 197
  14. 2019Commonsense Knowledge Mining from Pretrained Modelscited by 189
  15. 2021How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVENcited by 124
  16. 2021Large-Scale Differentially Private BERTcited by 118
  17. 2020Revisiting Membership Inference Under Realistic Assumptionscited by 112
  18. 2022Memorization in NLP Fine-tuning Methodscited by 45
  19. 2021Efficient (Soft) Q-Learning for Text Generation with Limited Good Datacited by 37
  20. 2022Text Revealer: Private Text Reconstruction via Model Inversion Attacks against Transformerscited by 31
  21. 2021How BPE Affects Memorization in Transformerscited by 29

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 19; 2026: 13.

32 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  2. 2025Riddle Me This! Stealthy Membership Inference for Retrieval-Augmented Generationcited by 26
  3. 2025Integrating LLMs with ITS: Recent Advances, Potentials, Challenges, and Future Directionscited by 23
  4. 2025Privacy Attacks on Image AutoRegressive Modelscited by 15
  5. 2025Synthetic Data Privacy Metricscited by 8
  6. 2025Personalized Language Models via Privacy-Preserving Evolutionary Model Mergingcited by 8
  7. 2025Federated In-Context Learning: Iterative Refinement for Improved Answer Qualitycited by 8
  8. 2025Unlocking Post-hoc Dataset Inference with Synthetic Datacited by 7
  9. 2025A Survey on Collaborating Small and Large Language Models for Performance, Cost-effectiveness, Cloud-edge Privacy, and Trustworthinesscited by 7
  10. 2025SoK: Data Reconstruction Attacks Against Machine Learning Models: Definition, Metrics, and Benchmarkcited by 5
  11. 2025Towards Harnessing the Collaborative Power of Large and Small Models for Domain Taskscited by 4
  12. 2025ContextLeak: Auditing Leakage in Private In-Context Learning Methodscited by 4
20 more
  1. 2026Benchmarking Empirical Privacy Protection for Adaptations of Large Language Modelscited by 4
  2. 2025Securing RAG: A Risk Assessment and Mitigation Frameworkcited by 3
  3. 2025ACTG-ARL: Differentially Private Conditional Text Generation with RL-Boosted Controlcited by 3
  4. 2025Privacy Risks and Preservation Methods in Explainable Artificial Intelligence: A Scoping Reviewcited by 2
  5. 2026Differentially Private Retrieval-Augmented Generationcited by 2
  6. 2025Efficient and Privacy-Preserving Soft Prompt Transfer for LLMscited by 1
  7. 2025Differentially Private In-Context Learning with Nearest Neighbor Searchcited by 1
  8. 2026Differentially Private Multimodal In-Context Learningcited by 1
  9. 2026Privacy Vulnerabilities of Attention Layers in Tabular Foundation Models and Protection of High-Risk Queriescited by 1
  10. 2026Probing Memorization of Tabular In-Context Learningcited by 1
  11. 2025DP-GPL: Differentially Private Graph Prompt Learningcited by 0
  12. 2025Membership Inference Attacks on In-Context Examples in LLM-based Recommender Systemscited by 0
  13. 2026Synapse: Federated Tool Routing via Typed Compendium Artifactscited by 0
  14. 2026Private PoEtry: Private In-Context Learning via Product of Expertscited by 0
  15. 2026Membership Inference Attacks for Retrieval Based In-Context Learning for Document Question Answeringcited by 0
  16. 2026Privacy Without Losing Place: A Paradigm for Private Retrieval in Spatial RAGscited by 0
  17. 2026PrivBench: A Holistic and Modular Benchmarking Platform for Evaluating Text-to-Text Privatizationcited by 0
  18. 2026Context Inference Attacks Without Jailbreakscited by 0
  19. 2026"Nothing to See Here'': Unintended Disclosure through Revision Traces of LLM Deliverablescited by 0
  20. 2026PFedMA: Who Should Join Next as Federated Multi-Agent Orchestration Evolves?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Membership Inference Attacks Against In-Context Learning
  2. 2025A Survey on Privacy Risks and Protection in Large Language Models
  3. 2025On Membership Inference Attacks in Knowledge Distillation
  4. 2025Membership Inference Attacks on Large-Scale Models: A Survey
  5. 2024SoK: Reducing the Vulnerability of Fine-tuned Language Models to Membership Inference Attacks
  6. 2023Last One Standing: A Comparative Analysis of Security and Privacy of Soft Prompt Tuning, LoRA, and In-Context Learning
  7. 2025Public Data Assisted Differentially Private In-Context Learning
  8. 2024Privacy-Preserving Large Language Models: Mechanisms, Applications, and Future Directions