mlsanityData to 7 Oct 2026

Label-Only Membership Inference Attacks

arXiv 2007.14321 ↗First posted 28 Jul 2020cs.CR cs.LG stat.MLcited by 371

Not on the map. Why?

Cited by 183 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Membership inference attacks are one of the simplest forms of privacy leakage for machine learning models: given a data point and model, determine whether the point was used to train the model. Existing membership inference attacks exploit models' abnormal confidence when queried on their training data. These attacks do not apply if the adversary only gets access to models' predicted labels, without a confidence measure. In this paper, we introduce label-only membership inference attacks. Instead of relying on confidence scores, our attacks evaluate the robustness of a model's predicted labels under perturbations to obtain a fine-grained membership signal. These perturbations include common data augmentations or adversarial examples. We empirically show that our label-only membership inference attacks perform on par with prior attacks that required access to model confidences. We further demonstrate that label-only attacks break multiple defenses against membership inference attacks that (implicitly or explicitly) rely on a phenomenon we call confidence masking. These defenses modify a model's confidence scores in order to thwart attacks, but leave the model's predicted labels unchanged. Our label-only attacks demonstrate that confidence-masking is not a viable defense strategy against membership inference. Finally, we investigate worst-case label-only attacks, that infer membership for a small number of outlier data points. We show that label-only attacks also match confidence-based attacks in this setting. We find that training models with differential privacy and (strong) L2 regularization are the only known defense strategies that successfully prevents all attacks. This remains true even when the differential privacy budget is too high to offer meaningful provable guarantees.

Key facts. Cited by 371 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Jul 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 33; 2022: 78; 2023: 75; 2024: 85; 2025: 76; 2026: 24.

371 papers.

Sort:

By area:

  1. 2021Membership Inference Attacks From First Principlescited by 839
  2. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  3. 2021Membership Inference Attacks on Machine Learning: A Surveycited by 306
  4. 2023Membership Inference Attacks against Language Models via Neighbourhood Comparisoncited by 249
  5. 2021Enhanced Membership Inference Attacks against Machine Learning Modelscited by 237
  6. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 218
  7. 2021Adversary Instantiation: Lower Bounds for Differentially Private Machine Learningcited by 193
  8. 2021On the Importance of Difficulty Calibration in Membership Inference Attackscited by 137
  9. 2023Are Diffusion Models Vulnerable to Membership Inference Attacks?cited by 135
  10. 2022The Privacy Onion Effect: Memorization is Relativecited by 128
  11. 2022Privacy for Free: How does Dataset Condensation Help Privacy?cited by 119
  12. 2021Dataset Inference: Ownership Resolution in Machine Learningcited by 115
38 more
  1. 2025The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injectionscited by 108
  2. 2023Low-Cost High-Power Membership Inference Attackscited by 99
  3. 2022Membership Inference Attacks by Exploiting Loss Trajectorycited by 93
  4. 2023Practical Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibrationcited by 90
  5. 2021Node-Level Membership Inference Attacks Against Graph Neural Networkscited by 80
  6. 2021Bad Characters: Imperceptible NLP Attackscited by 69
  7. 2021EncoderMI: Membership Inference against Pre-trained Encoders in Contrastive Learningcited by 68
  8. 2023Bag of Tricks for Training Data Extraction from Language Modelscited by 67
  9. 2022Membership Inference Attacks Against Text-to-image Generation Modelscited by 66
  10. 2023Learning to Unlearn: Instance-wise Unlearning for Pre-trained Classifierscited by 66
  11. 2021Mitigating Membership Inference Attacks by Self-Distillation Through a Novel Ensemble Architecturecited by 63
  12. 2023Differentially Private Synthetic Data via Foundation Model APIs 1: Imagescited by 63
  13. 2022Plug & Play Attacks: Towards Robust and Flexible Model Inversion Attackscited by 58
  14. 2021Property Inference From Poisoningcited by 56
  15. 2022Label-Only Model Inversion Attacks via Boundary Repulsioncited by 56
  16. 2024SoK: Membership Inference Attacks on LLMs are Rushing Nowhere (and How to Fix It)cited by 54
  17. 2021Membership Inference Attacks Against Recommender Systemscited by 52
  18. 2022Exploring the Limits of Differentially Private Deep Learning with Group-wise Clippingcited by 52
  19. 2021Adapting Membership Inference Attacks to GNN for Graph Classification: Approaches and Implicationscited by 50
  20. 2021Model Stealing Attacks Against Inductive Graph Neural Networkscited by 50
  21. 2023Prompt Stealing Attacks Against Text-to-Image Generation Modelscited by 48
  22. 2023Exposing the Fake: Effective Diffusion-Generated Images Detectioncited by 48
  23. 2022SoK: Let the Privacy Games Begin! A Unified Treatment of Data Inference Privacy in Machine Learningcited by 47
  24. 2023Membership Inference of Diffusion Modelscited by 46
  25. 2021Survey: Leakage and Privacy at Inference Timecited by 44
  26. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  27. 2023Training Data Extraction From Pre-trained Language Models: A Surveycited by 43
  28. 2022Membership Inference Attacks and Defenses in Neural Network Pruningcited by 42
  29. 2023White-box Membership Inference Attacks against Diffusion Modelscited by 42
  30. 2022Large Scale Transfer Learning for Differentially Private Image Classificationcited by 40
  31. 2022Towards A Proactive ML Approach for Detecting Backdoor Poison Samplescited by 40
  32. 2023Re-thinking Model Inversion Attacks Against Deep Neural Networkscited by 39
  33. 2023Privacy Side Channels in Machine Learning Systemscited by 38
  34. 2024Membership Inference Attacks Against In-Context Learningcited by 38
  35. 2023Practical Membership Inference Attacks Against Large-Scale Multi-Modal Models: A Pilot Studycited by 37
  36. 2023Learn to Unlearn for Deep Neural Networks: Minimizing Unlearning Interference with Gradient Projectioncited by 37
  37. 2022Canary in a Coalmine: Better Membership Inference with Ensembled Adversarial Queriescited by 36
  38. 2023User Inference Attacks on Large Language Modelscited by 36

50 most cited of 371 All 371 (JSON)