mlsanityData to 7 Oct 2026

Do Membership Inference Attacks Work on Large Language Models?

arXiv 2402.07841 ↗First posted 12 Feb 2024cs.CLcited by 188

On the map: LLM safety and security → Membership inference attacks

Abstract (arXiv)

Membership inference attacks (MIAs) attempt to predict whether a particular datapoint is a member of a target model's training data. Despite extensive research on traditional machine learning models, there has been limited work studying MIA on the pre-training data of large language models (LLMs). We perform a large-scale evaluation of MIAs over a suite of language models (LMs) trained on the Pile, ranging from 160M to 12B parameters. We find that MIAs barely outperform random guessing for most settings across varying LLM sizes and domains. Our further analyses reveal that this poor performance can be attributed to (1) the combination of a large dataset and few training iterations, and (2) an inherently fuzzy boundary between members and non-members. We identify specific settings where LLMs have been shown to be vulnerable to membership inference and show that the apparent success in such settings can be attributed to a distribution shift, such as when members and non-members are drawn from the seemingly identical domain but with different temporal ranges. We release our code and data as a unified benchmark package that includes all existing MIAs, supporting future work.

Key facts. Cited by 188 papers in the map (data through 7 Oct 2026). Placed in Membership inference attacks, one of 12 directions in LLM safety and security. Builds on 49 of its 70 references that are in the map. First posted on arXiv 12 Feb 2024.

Builds on

49 of 70 references.

Closest ancestors · 11

Same direction, newest first

  1. 2023Practical Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibration10 Nov
  2. 2023Did the Neurons Read your Book? Document-level Membership Inference for Large Language Models23 Oct
  3. 2023Scalable Membership Inference Attacks via Quantile Regression7 Jul
  4. 2023Membership Inference Attacks against Language Models via Neighbourhood Comparison29 May
  5. 2023Analyzing Leakage of Personally Identifiable Information in Language Models1 Feb
  6. 2022RelaxLoss: Defending Membership Inference Attacks without Losing Utility12 Jul
  7. 2022Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks8 Mar
  8. 2021Membership Inference Attacks From First Principles7 Dec
  9. 2021Enhanced Membership Inference Attacks against Machine Learning Models18 Nov
  10. 2021On the Importance of Difficulty Calibration in Membership Inference Attacks15 Nov
  11. 2021Mitigating Membership Inference Attacks by Self-Distillation Through a Novel Ensemble Architecture15 Oct

Other references · 38

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2016Deep Learning with Differential Privacycited by 2,682
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  8. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  9. 2020Extracting Training Data from Large Language Modelscited by 2,144
  10. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  11. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  12. 2022Quantifying Memorization Across Neural Language Modelscited by 755
26 more
  1. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  2. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  3. 2024OLMo: Accelerating the Science of Language Modelscited by 599
  4. 2024Dolma: an Open Corpus of Three Trillion Tokens for Language Model Pretraining Researchcited by 475
  5. 2023Scaling Data-Constrained Language Modelscited by 391
  6. 2023Detecting Pretraining Data from Large Language Modelscited by 375
  7. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 309
  8. 2019White-box vs Black-box: Bayes Optimal Strategies for Membership Inferencecited by 265
  9. 2023Proving Test Set Contamination in Black Box Language Modelscited by 227
  10. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 218
  11. 2023Rethinking Benchmark and Contamination for Language Models with Rephrased Samplescited by 192
  12. 2023A Pretrainer's Guide to Training Data: Measuring the Effects of Data Age, Domain Coverage, Quality, & Toxicitycited by 180
  13. 2023Emergent and Predictable Memorization in Large Language Modelscited by 147
  14. 2023Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4cited by 123
  15. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
  16. 2023Privacy Auditing with One (1) Training Runcited by 119
  17. 2022Measuring Forgetting of Memorized Training Examplescited by 118
  18. 2021Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?cited by 91
  19. 2023SILO Language Models: Isolating Legal Risk In a Nonparametric Datastorecited by 71
  20. 2023Paloma: A Benchmark for Evaluating Language Model Fitcited by 53
  21. 2021Formalizing and Estimating Distribution Inference Riskscited by 42
  22. 2019One Epoch Is All You Needcited by 41
  23. 2023User Inference Attacks on Large Language Modelscited by 36
  24. 2023Advancing Differential Privacy: Where We Are Now and Future Directions for Real-World Deploymentcited by 26
  25. 2023MoPe: Model Perturbation-based Privacy Attacks on Language Modelscited by 25
  26. 2020KART: Parameterization of Privacy Leakage Scenarios from Pre-trained Language Modelscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 46; 2025: 85; 2026: 57.

188 papers.

Sort:

By area:

  1. 2024Rethinking Machine Unlearning for Large Language Modelscited by 247
  2. 2024SecAlign: Defending Against Prompt Injection with Preference Optimizationcited by 179
  3. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  4. 2024Large Language Model Unlearning via Embedding-Corrupted Promptscited by 112
  5. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  6. 2024LLM Dataset Inference: Did you train on my dataset?cited by 107
  7. 2024Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Modelscited by 102
  8. 2024Rethinking LLM Memorization through the Lens of Adversarial Compressioncited by 91
  9. 2024AirGapAgent: Protecting Privacy-Conscious Conversational Agentscited by 90
  10. 2025OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metricscited by 80
  11. 2025AgentDAM: Privacy Leakage Evaluation for Autonomous Web Agentscited by 79
  12. 2024RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Modelscited by 77
38 more
  1. 2024LLM Unlearning via Loss Adjustment with Only Forget Datacited by 71
  2. 2024Blind Baselines Beat Membership Inference Attacks for Foundation Modelscited by 65
  3. 2025Rethinking LLM Unlearning Objectives: A Gradient Perspective and Go Beyondcited by 59
  4. 2024SoK: Membership Inference Attacks on LLMs are Rushing Nowhere (and How to Fix It)cited by 54
  5. 2025How much do language models memorize?cited by 53
  6. 2024Membership Inference Attacks Cannot Prove that a Model Was Trained On Your Datacited by 49
  7. 2024Is My Data in Your Retrieval Database? Membership Inference Attacks Against Retrieval Augmented Generationcited by 44
  8. 2025Unlearning Isn't Deletion: Investigating Reversibility of Machine Unlearning in LLMscited by 41
  9. 2024Elephants Never Forget: Memorization and Learning of Tabular Data in Large Language Modelscited by 34
  10. 2025Towards Label-Only Membership Inference Attack against Pre-trained Large Language Modelscited by 34
  11. 2024Alpaca against Vicuna: Using LLMs to Uncover Memorization of LLMscited by 33
  12. 2024Membership Inference Attacks against Large Vision-Language Modelscited by 31
  13. 2024Auditing f-Differential Privacy in One Runcited by 28
  14. 2024Towards Effective Evaluations and Comparisons for LLM Unlearning Methodscited by 27
  15. 2025Membership Inference Attacks Against Vision-Language Modelscited by 26
  16. 2024Generating Is Believing: Membership Inference Attacks against Retrieval-Augmented Generationcited by 25
  17. 2024Exposing Privacy Gaps: Membership Inference Attack on Preference Data for LLM Alignmentcited by 24
  18. 2025Privacy Auditing of Large Language Modelscited by 22
  19. 2024Generated Data with Fake Privacy: Hidden Dangers of Fine-tuning Large Language Models on Generated Datacited by 21
  20. 2024Learning on LoRAs: GL-Equivariant Processing of Low-Rank Weight Spaces for Large Finetuned Modelscited by 21
  21. 2024Context-Aware Membership Inference Attacks against Pre-trained Large Language Modelscited by 20
  22. 2024When Machine Unlearning Meets Retrieval-Augmented Generation (RAG): Keep Secret or Forget Knowledge?cited by 20
  23. 2025Language Models May Verbatim Complete Text They Were Not Explicitly Trained Oncited by 20
  24. 2025Reality Check: A New Evaluation Ecosystem Is Necessary to Understand AI's Real World Effectscited by 18
  25. 2024Semantic Membership Inference Attack against Large Language Modelscited by 17
  26. 2025Membership Inference Attacks on Large-Scale Models: A Surveycited by 17
  27. 2025GUARD: Generation-time LLM Unlearning via Adaptive Restriction and Detectioncited by 17
  28. 2024Ward: Provable RAG Dataset Inference via LLM Watermarkscited by 15
  29. 2025Privacy Attacks on Image AutoRegressive Modelscited by 15
  30. 2025Position: Adversarial ML for LLMs Is Not Making Any Progresscited by 14
  31. 2025Doxing via the Lens: Revealing Location-related Privacy Leakage on Multi-modal Large Reasoning Modelscited by 14
  32. 2024Textual Unlearning Gives a False Sense of Unlearningcited by 13
  33. 2024Data Mixture Inference: What do BPE Tokenizers Reveal about their Training Data?cited by 13
  34. 2024Detecting Training Data of Large Language Models via Expectation Maximizationcited by 13
  35. 2025How Contaminated Is Your Benchmark? Quantifying Dataset Leakage in Large Language Models with Kernel Divergencecited by 13
  36. 2024A Comprehensive Survey of Contamination Detection Methods in Large Language Modelscited by 12
  37. 2024Measuring memorization in RLHF for code completioncited by 12
  38. 2025Reviewing Scientific Papers for Critical Problems With Reasoning LLMs: Baseline Approaches and Automatic Evaluationcited by 12

50 most cited of 188 All 188 (JSON)