mlsanityData to 7 Oct 2026

Membership Inference Attacks against Language Models via Neighbourhood Comparison

arXiv 2305.18462 ↗First posted 29 May 2023cs.CL cs.CR cs.LGcited by 249

On the map: LLM safety and security → Membership inference attacks

Abstract (arXiv)

Membership Inference attacks (MIAs) aim to predict whether a data sample was present in the training data of a machine learning model or not, and are widely used for assessing the privacy risks of language models. Most existing attacks rely on the observation that models tend to assign higher probabilities to their training samples than non-training points. However, simple thresholding of the model score in isolation tends to lead to high false-positive rates as it does not account for the intrinsic complexity of a sample. Recent work has demonstrated that reference-based attacks which compare model scores to those obtained from a reference model trained on similar data can substantially improve the performance of MIAs. However, in order to train reference models, attacks of this kind make the strong and arguably unrealistic assumption that an adversary has access to samples closely resembling the original training data. Therefore, we investigate their performance in more realistic scenarios and find that they are highly fragile in relation to the data distribution used to train reference models. To investigate whether this fragility provides a layer of safety, we propose and evaluate neighbourhood attacks, which compare model scores for a given sample to scores of synthetically generated neighbour texts and therefore eliminate the need for access to the training data distribution. We show that, in addition to being competitive with reference-based attacks that have perfect knowledge about the training data distribution, our attack clearly outperforms existing reference-free attacks as well as reference-based attacks with imperfect knowledge, which demonstrates the need for a reevaluation of the threat model of adversarial attacks.

Key facts. Cited by 249 papers in the map (data through 7 Oct 2026). Placed in Membership inference attacks, one of 12 directions in LLM safety and security. Builds on 39 of its 50 references that are in the map. First posted on arXiv 29 May 2023.

Builds on

39 of 50 references.

Closest ancestors · 6

Same direction, newest first

  1. 2022Membership Inference Attacks by Exploiting Loss Trajectory31 Aug
  2. 2022RelaxLoss: Defending Membership Inference Attacks without Losing Utility12 Jul
  3. 2022Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attacks8 Mar
  4. 2021Membership Inference Attacks From First Principles7 Dec
  5. 2021Enhanced Membership Inference Attacks against Machine Learning Models18 Nov
  6. 2021On the Importance of Difficulty Calibration in Membership Inference Attacks15 Nov

Other references · 33

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2016Pointer Sentinel Mixture Modelscited by 2,992
  5. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  6. 2016Deep Learning with Differential Privacycited by 2,682
  7. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  8. 2020Extracting Training Data from Large Language Modelscited by 2,144
  9. 2018Hierarchical Neural Story Generationcited by 1,182
  10. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  11. 2023DetectGPT: Zero-Shot Machine-Generated Text Detection using Probability Curvaturecited by 705
  12. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
21 more
  1. 2023Paraphrasing evades detectors of AI-generated text, but retrieval is an effective defensecited by 401
  2. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  3. 2020Label-Only Membership Inference Attackscited by 371
  4. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 309
  5. 2019White-box vs Black-box: Bayes Optimal Strategies for Membership Inferencecited by 265
  6. 2020Information Leakage in Embedding Modelscited by 199
  7. 2019MemGuard: Defending against Black-Box Membership Inference Attacks via Adversarial Examplescited by 183
  8. 2019Stolen Memories: Leveraging Model Memorization for Calibrated White-Box Membership Inferencecited by 168
  9. 2020Membership Leakage in Label-Only Exposurescited by 156
  10. 2021Counterfactual Memorization in Neural Language Modelscited by 155
  11. 2022The Privacy Onion Effect: Memorization is Relativecited by 128
  12. 2020Revisiting Membership Inference Under Realistic Assumptionscited by 108
  13. 2023On the Possibilities of AI-Generated Text Detectioncited by 88
  14. 2022Position: Considerations for Differentially Private Learning with Large-Scale Public Pretrainingcited by 86
  15. 2018Understanding Membership Inferences on Well-Generalized Learning Modelscited by 83
  16. 2019Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?cited by 68
  17. 2020ML Privacy Meter: Aiding Regulatory Compliance by Quantifying the Privacy Risks of Machine Learningcited by 66
  18. 2022Memorization in NLP Fine-tuning Methodscited by 43
  19. 2023Smaller Language Models are Better Black-box Machine-Generated Text Detectorscited by 35
  20. 2021Privacy Regularization: Joint Privacy-Utility Optimization in Language Modelscited by 29
  21. 2022Differentially Private Model Compressioncited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 18; 2024: 90; 2025: 87; 2026: 54.

249 papers.

Sort:

By area:

  1. 2023Detecting Pretraining Data from Large Language Modelscited by 375
  2. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agentscited by 341
  3. 2023Formalizing and Benchmarking Prompt Injection Attacks and Defensescited by 328
  4. 2024PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Modelscited by 251
  5. 2023Proving Test Set Contamination in Black Box Language Modelscited by 227
  6. 2024Do Membership Inference Attacks Work on Large Language Models?cited by 188
  7. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 179
  8. 2024SecAlign: Defending Against Prompt Injection with Preference Optimizationcited by 179
  9. 2024PrivacyLens: Evaluating Privacy Norm Awareness of Language Models in Actioncited by 161
  10. 2024LLM Dataset Inference: Did you train on my dataset?cited by 107
  11. 2024Best Practices and Lessons Learned on Synthetic Datacited by 104
  12. 2024Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Modelscited by 102
38 more
  1. 2024Investigating Data Contamination for Pre-training Language Modelscited by 91
  2. 2023Practical Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibrationcited by 90
  3. 2024What makes unlearning hard and what to do about itcited by 89
  4. 2024Fast Adversarial Attacks on Language Models In One GPU Minutecited by 85
  5. 2024Don't Listen To Me: Understanding and Exploring Jailbreak Prompts of Large Language Modelscited by 77
  6. 2024Differentially Private Synthetic Data via Foundation Model APIs 2: Textcited by 74
  7. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 63
  8. 2024ReCaLL: Membership Inference via Relative Conditional Log-Likelihoodscited by 61
  9. 2024SoK: Membership Inference Attacks on LLMs are Rushing Nowhere (and How to Fix It)cited by 54
  10. 2024Pretraining Data Detection for Large Language Models: A Divergence-based Calibration Methodcited by 54
  11. 2023Did the Neurons Read your Book? Document-level Membership Inference for Large Language Modelscited by 51
  12. 2023Privacy in Large Language Models: Attacks, Defenses and Future Directionscited by 48
  13. 2024Dated Data: Tracing Knowledge Cutoffs in Large Language Modelscited by 48
  14. 2024Phantom: General Backdoor Attacks on Retrieval Augmented Language Generationcited by 46
  15. 2025VeriGuard: Enhancing LLM Agent Safety via Verified Code Generationcited by 46
  16. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  17. 2024To Each (Textual Sequence) Its Own: Improving Memorized-Data Unlearning in Large Language Modelscited by 42
  18. 2024LLM-PBE: Assessing Data Privacy in Large Language Modelscited by 38
  19. 2024Membership Inference Attacks Against In-Context Learningcited by 38
  20. 2023User Inference Attacks on Large Language Modelscited by 36
  21. 2023SoK: Memorization in General-Purpose Large Language Modelscited by 36
  22. 2024Copyright Traps for Large Language Modelscited by 36
  23. 2024Elephants Never Forget: Memorization and Learning of Tabular Data in Large Language Modelscited by 34
  24. 2025Towards Label-Only Membership Inference Attack against Pre-trained Large Language Modelscited by 34
  25. 2024ConStat: Performance-Based Contamination Detection in Large Language Modelscited by 33
  26. 2023Identifying and Mitigating Privacy Risks Stemming from Language Models: A Surveycited by 30
  27. 2024Evading Data Contamination Detection for Language Models is (too) Easycited by 30
  28. 2024Localizing Paragraph Memorization in Language Modelscited by 30
  29. 2024Unveiling the Spectrum of Data Contamination in Language Models: A Survey from Detection to Remediationcited by 30
  30. 2024Undesirable Memorization in Large Language Models: A Surveycited by 29
  31. 2024Mask-based Membership Inference Attacks for Retrieval-Augmented Generationcited by 29
  32. 2023Integration of Large Language Models and Federated Learningcited by 28
  33. 2023DPZero: Private Fine-Tuning of Language Models without Backpropagationcited by 26
  34. 2024Does Data Contamination Detection Work (Well) for LLMs? A Survey and Evaluation on Detection Assumptionscited by 26
  35. 2025Riddle Me This! Stealthy Membership Inference for Retrieval-Augmented Generationcited by 26
  36. 2023MoPe: Model Perturbation-based Privacy Attacks on Language Modelscited by 25
  37. 2024Generating Is Believing: Membership Inference Attacks against Retrieval-Augmented Generationcited by 25
  38. 2025AgentSentinel: An End-to-End and Real-Time Security Defense Framework for Computer-Use Agentscited by 25

50 most cited of 249 All 249 (JSON)