mlsanityData to 14 Oct 2026

Membership Inference Attacks Against In-Context Learning

arXiv 2409.01380 ↗First posted 2 Sep 2024cs.CR cs.CLcited by 43

On the map: LLM safety and security → Membership inference attacks

Abstract (arXiv)

Adapting Large Language Models (LLMs) to specific tasks introduces concerns about computational efficiency, prompting an exploration of efficient methods such as In-Context Learning (ICL). However, the vulnerability of ICL to privacy attacks under realistic assumptions remains largely unexplored. In this work, we present the first membership inference attack tailored for ICL, relying solely on generated texts without their associated probabilities. We propose four attack strategies tailored to various constrained scenarios and conduct extensive experiments on four popular large language models. Empirical results show that our attacks can accurately determine membership status in most cases, e.g., 95% accuracy advantage against LLaMA, indicating that the associated risks are much higher than those shown by existing probability-based attacks. Additionally, we propose a hybrid attack that synthesizes the strengths of the aforementioned strategies, achieving an accuracy advantage of over 95% in most cases. Furthermore, we investigate three potential defenses targeting data, instruction, and output. Results demonstrate combining defenses from orthogonal dimensions significantly reduces privacy leakage and offers enhanced privacy assurances.

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). Placed in Membership inference attacks, one of 12 directions in LLM safety and security. Builds on 37 of its 49 references that are in the map. First posted on arXiv 2 Sep 2024.

Builds on

37 of 49 references.

Closest ancestors · 10

Same direction, newest first

  1. 2023Practical Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibration10 Nov
  2. 2023Last One Standing: A Comparative Analysis of Security and Privacy of Soft Prompt Tuning, LoRA, and In-Context Learning17 Oct
  3. 2023Membership Inference Attacks against Language Models via Neighbourhood Comparison29 May
  4. 2022Membership Inference Attacks Against Text-to-image Generation Models3 Oct
  5. 2022Membership Inference Attacks by Exploiting Loss Trajectory31 Aug
  6. 2022Membership-Doctor: Comprehensive Assessment of Membership Inference Against Machine Learning Models22 Aug
  7. 2021Membership Inference Attacks From First Principles7 Dec
  8. 2021Enhanced Membership Inference Attacks against Machine Learning Models18 Nov
  9. 2021Membership Inference Attacks on Machine Learning: A Survey14 Mar
  10. 2021ML-Doctor: Holistic Risk Assessment of Inference Attacks Against Machine Learning Models4 Feb

Other references · 27

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  6. 2015Character-level Convolutional Networks for Text Classificationcited by 2,330
  7. 2020Extracting Training Data from Large Language Modelscited by 2,189
  8. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,392
  9. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,045
  10. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
  11. 2018ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Modelscited by 498
  12. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 392
15 more
  1. 2020Label-Only Membership Inference Attackscited by 388
  2. 2023Detecting Pretraining Data from Large Language Modelscited by 387
  3. 2020Systematic Evaluation of Privacy Risks of Machine Learning Modelscited by 292
  4. 2019Stolen Memories: Leveraging Model Memorization for Calibrated White-Box Membership Inferencecited by 175
  5. 2020Membership Leakage in Label-Only Exposurescited by 168
  6. 2019Privacy Risks of Securing Machine Learning Models against Adversarial Examplescited by 141
  7. 2023Is ChatGPT a Good Sentiment Analyzer? A Preliminary Studycited by 96
  8. 2023Privacy-Preserving In-Context Learning with Differentially Private Few-Shot Generationcited by 96
  9. 2021Node-Level Membership Inference Attacks Against Graph Neural Networkscited by 84
  10. 2024Instruction Backdoor Attacks Against Customized LLMscited by 62
  11. 2022SoK: Let the Privacy Games Begin! A Unified Treatment of Data Inference Privacy in Machine Learningcited by 51
  12. 2023Multi-Dimensional Evaluation of Text Summarization with In-Context Learningcited by 44
  13. 2022Auditing Membership Leakages of Multi-Exit Networkscited by 27
  14. 2023SummQA at MEDIQA-Chat 2023:In-Context Learning with GPT-4 for Medical Summarizationcited by 13
  15. 2023The Scope of In-Context Learning for the Extraction of Medical Temporal Constraintscited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 2; 2025: 18; 2026: 23.

43 papers.

Sort:

By area:

  1. 2025Differentially Private Steering for Large Language Model Alignmentcited by 15
  2. 2025Safeguarding Privacy of Retrieval Data against Membership Inference Attacks: Is This Query Too Close to Home?cited by 9
  3. 2025IP Leakage Attacks Targeting LLM-Based Multi-Agent Systemscited by 8
  4. 2025DCMI: A Differential Calibration Membership Inference Attack Against Retrieval-Augmented Generationcited by 7
  5. 2024Understanding Data Importance in Machine Learning Attacks: Does Valuable Data Pose Greater Harm?cited by 5
  6. 2025Peering Behind the Shield: Guardrail Identification in Large Language Modelscited by 5
  7. 2025SoK: Data Reconstruction Attacks Against Machine Learning Models: Definition, Metrics, and Benchmarkcited by 5
  8. 2025How do we measure privacy in text? A survey of text anonymization metricscited by 5
  9. 2026Membership Inference Attacks Against Fine-tuned Diffusion Language Modelscited by 5
  10. 2025Pattern Enhanced Multi-Turn Jailbreaking: Exploiting Structural Vulnerabilities in Large Language Modelscited by 4
  11. 2025Black-Box Membership Inference Attack for LVLMs via Prior Knowledge-Calibrated Memory Probingcited by 4
  12. 2025ContextLeak: Auditing Leakage in Private In-Context Learning Methodscited by 4
31 more
  1. 2026Benchmark of Benchmarks: Unpacking Influence and Code Repository Quality in LLM Safety Benchmarkscited by 4
  2. 2026Towards Secure Retrieval-Augmented Generation: A Comprehensive Review of Threats, Defenses and Benchmarkscited by 4
  3. 2026Five Queries Are Enough: Query-Efficient and Surrogate-Free Membership Inference Attacks on RAG via Entailmentcited by 4
  4. 2025SoK: Semantic Privacy in Large Language Modelscited by 3
  5. 2025CompLeak: Deep Learning Model Compression Exacerbates Privacy Leakagecited by 3
  6. 2025Membership Inference Attacks on Recommender System: A Surveycited by 3
  7. 2025Auditing Data Provenance in Real-world Text-to-Image Diffusion Models for Privacy and Copyright Protectioncited by 2
  8. 2025External Data Extraction Attacks against Retrieval-Augmented Large Language Modelscited by 2
  9. 2025Are Neuro-Inspired Multi-Modal Vision-Language Models Resilient to Membership Inference Privacy Leakage?cited by 2
  10. 2026LLM-enabled Applications Require System-Level Threat Monitoringcited by 2
  11. 2026Membership Inference Attacks against Large Audio Language Modelscited by 2
  12. 2024Membership Inference Attack against Long-Context Large Language Modelscited by 1
  13. 2025Public Data Assisted Differentially Private In-Context Learningcited by 1
  14. 2025Differentially Private In-Context Learning with Nearest Neighbor Searchcited by 1
  15. 2026Leveraging Soft Prompts for Privacy Attacks in Federated Prompt Tuningcited by 1
  16. 2026United We Defend: Collaborative Membership Inference Defenses in Federated Learningcited by 1
  17. 2026SoK: Challenges in Tabular Membership Inference Attackscited by 1
  18. 2026WinFLoRA: Incentivizing Client-Adaptive Aggregation in Federated LoRA under Privacy Heterogeneitycited by 1
  19. 2026Differentially Private Multimodal In-Context Learningcited by 1
  20. 2026Privacy Guard & Token Parsimony by Prompt and Context Handling and LLM Routingcited by 1
  21. 2026Privacy Vulnerabilities of Attention Layers in Tabular Foundation Models and Protection of High-Risk Queriescited by 1
  22. 2026Probing Memorization of Tabular In-Context Learningcited by 1
  23. 2026AutoMIA: Improved Baselines for Membership Inference Attack via Agentic Self-Explorationcited by 0
  24. 2026Membership Inference Attacks for Retrieval Based In-Context Learning for Document Question Answeringcited by 0
  25. 2026Pop Quiz Attack: Black-box Membership Inference Attacks Against Large Language Modelscited by 0
  26. 2026Securing Code Understanding: Detecting Natural Backdoor Vulnerability in Code Language Modelscited by 0
  27. 2026Have You Ever Seen Them? Entity-level Membership Inference through Interrogating Large Language Modelscited by 0
  28. 2026When T2I Synthetic Data Backfires: Amplified Privacy Risks in Real-Synthetic Mix Trainingcited by 0
  29. 2026Context Inference Attacks Without Jailbreakscited by 0
  30. 2026"Nothing to See Here'': Unintended Disclosure through Revision Traces of LLM Deliverablescited by 0
  31. 2026SoK: Rethinking Membership Inference in Tabular Machine Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024On the Privacy Risk of In-context Learning
  2. 2024Order of Magnitude Speedups for LLM Membership Inference
  3. 2025On Membership Inference Attacks in Knowledge Distillation
  4. 2025Membership Inference Attacks on Large-Scale Models: A Survey
  5. 2022Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks
  6. 2024SoK: Reducing the Vulnerability of Fine-tuned Language Models to Membership Inference Attacks
  7. 2025Tokens for Learning, Tokens for Unlearning: Mitigating Membership Inference Attacks in Large Language Models via Dual-Purpose Training
  8. 2025In-Context Probing for Membership Inference in Fine-Tuned Language Models