mlsanityData to 7 Oct 2026

Evaluations of Machine Learning Privacy Defenses are Misleading

arXiv 2404.17399 ↗First posted 26 Apr 2024cs.CR cs.LGcited by 35

Not on the map. Why?

Abstract (arXiv)

Empirical defenses for machine learning privacy forgo the provable guarantees of differential privacy in the hope of achieving higher utility while resisting realistic adversaries. We identify severe pitfalls in existing empirical privacy evaluations (based on membership inference attacks) that result in misleading conclusions. In particular, we show that prior evaluations fail to characterize the privacy leakage of the most vulnerable samples, use weak attacks, and avoid comparisons with practical differential privacy baselines. In 5 case studies of empirical privacy defenses, we find that prior evaluations underestimate privacy leakage by an order of magnitude. Under our stronger evaluation, none of the empirical defenses we study are competitive with a properly tuned, high-utility DP-SGD baseline (with vacuous provable guarantees).

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 52 of its 72 references that are in the map. First posted on arXiv 26 Apr 2024.

Builds on

52 of 72 references.

References · 52

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  4. 2016Wide Residual Networkscited by 3,057
  5. 2016Deep Learning with Differential Privacycited by 2,682
  6. 2020Extracting Training Data from Large Language Modelscited by 2,144
  7. 2018Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examplescited by 1,259
  8. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  9. 2021Membership Inference Attacks From First Principlescited by 839
  10. 2017Ensemble Adversarial Training: Attacks and Defensescited by 833
  11. 2020On Adaptive Attacks to Adversarial Example Defensescited by 547
  12. 2018ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Modelscited by 475
38 more
  1. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 452
  2. 2021Opacus: User-Friendly Differential Privacy Library in PyTorchcited by 375
  3. 2020Label-Only Membership Inference Attackscited by 371
  4. 2016Semi-supervised Knowledge Transfer for Deep Learning from Private Training Datacited by 347
  5. 2013The Composition Theorem for Differential Privacycited by 290
  6. 2019White-box vs Black-box: Bayes Optimal Strategies for Membership Inferencecited by 265
  7. 2021Enhanced Membership Inference Attacks against Machine Learning Modelscited by 237
  8. 2020Differentially Private Learning Needs Better Features (or Much More Data)cited by 233
  9. 2022Unlocking High-Accuracy Differentially Private Image Classification through Scalecited by 218
  10. 2020Auditing Differentially Private Machine Learning: How Private is Private SGD?cited by 213
  11. 2019Data-Free Learning of Student Networkscited by 203
  12. 2021Adversary Instantiation: Lower Bounds for Differentially Private Machine Learningcited by 193
  13. 2019MemGuard: Defending against Black-Box Membership Inference Attacks via Adversarial Examplescited by 183
  14. 2018Machine Learning with Membership Privacy using Adversarial Regularizationcited by 177
  15. 2021On the Importance of Difficulty Calibration in Membership Inference Attackscited by 137
  16. 2022The Privacy Onion Effect: Memorization is Relativecited by 128
  17. 2022Privacy for Free: How does Dataset Condensation Help Privacy?cited by 119
  18. 2023Privacy Auditing with One (1) Training Runcited by 119
  19. 2021DENSE: Data-Free One-Shot Federated Learningcited by 100
  20. 2023Low-Cost High-Power Membership Inference Attackscited by 99
  21. 2022Truth Serum: Poisoning Machine Learning Models to Reveal Their Secretscited by 94
  22. 2021EncoderMI: Membership Inference against Pre-trained Encoders in Contrastive Learningcited by 68
  23. 2021Mitigating Membership Inference Attacks by Self-Distillation Through a Novel Ensemble Architecturecited by 63
  24. 2022TAN Without a Burn: Scaling Laws of DP-SGDcited by 51
  25. 2023Towards More Realistic Membership Inference Attacks on Large Diffusion Modelscited by 41
  26. 2022Debugging Differential Privacy: A Case Study for Privacy Auditingcited by 39
  27. 2023Practical Membership Inference Attacks Against Large-Scale Multi-Modal Models: A Pilot Studycited by 37
  28. 2022Canary in a Coalmine: Better Membership Inference with Ensembled Adversarial Queriescited by 36
  29. 2021Quantifying and Mitigating Privacy Risks of Contrastive Learningcited by 35
  30. 2022RelaxLoss: Defending Membership Inference Attacks without Losing Utilitycited by 35
  31. 2020Defending Model Inversion and Membership Inference Attacks via Prediction Purificationcited by 32
  32. 2019Identifying Mislabeled Instances in Classification Datasetscited by 26
  33. 2023Overconfidence is a Dangerous Thing: Mitigating Membership Inference Attacks by Enforcing Less Confident Predictioncited by 26
  34. 2022No Free Lunch in "Privacy for Free: How does Dataset Condensation Help Privacy"cited by 23
  35. 2023Leave-one-out Distinguishability in Machine Learningcited by 20
  36. 2024Privacy Backdoors: Stealing Data with Corrupted Pretrained Modelscited by 20
  37. 2023Unleashing the Power of Randomization in Auditing Differentially Private MLcited by 19
  38. 2022On the Privacy Effect of Data Enhancement via the Lens of Memorizationcited by 14

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 9; 2025: 16; 2026: 10.

35 papers.

Sort:

By area:

  1. 2025The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injectionscited by 108
  2. 2024Membership Inference Attacks Cannot Prove that a Model Was Trained On Your Datacited by 49
  3. 2024Textual Unlearning Gives a False Sense of Unlearningcited by 13
  4. 2024One-shot Federated Learning via Synthetic Distiller-Distillate Communicationcited by 12
  5. 2025Does Training with Synthetic Data Truly Protect Privacy?cited by 12
  6. 2025Membership Inference Attacks as Privacy Tools: Reliability, Disparity and Ensemblecited by 11
  7. 2024The Last Iterate Advantage: Empirical Auditing and Principled Heuristic Analysis of Differentially Private SGDcited by 10
  8. 2024Underestimated Privacy Risks for Minority Populations in Large Language Model Unlearningcited by 9
  9. 2025How Well Can Differential Privacy Be Audited in One Run?cited by 8
  10. 2025Rectifying Privacy and Efficacy Measurements in Machine Unlearning: A New Inference Attack Perspectivecited by 8
  11. 2025Membership Inference Attacks on Sequence Modelscited by 6
  12. 2024Defending Membership Inference Attacks via Privacy-aware Sparsity Tuningcited by 4
23 more
  1. 2025Instance-Level Data-Use Auditing of Visual ML Modelscited by 4
  2. 2025The Hitchhiker's Guide to Efficient, End-to-End, and Tight DP Auditingcited by 4
  3. 2025Auditing Approximate Machine Unlearning for Differentially Private Modelscited by 4
  4. 2025Synthetic Data Can Mislead Evaluations: Membership Inference as Machine Text Detectioncited by 3
  5. 2025Membership Inference Attacks on Tokenizers of Large Language Modelscited by 3
  6. 2025Setting varepsilon is not the Issue in Differential Privacycited by 3
  7. 2026Revisiting the LiRA Membership Inference Attack Under Realistic Assumptionscited by 3
  8. 2024Trustworthiness of Stochastic Gradient Descent in Distributed Learningcited by 2
  9. 2025How does Bayesian Sampling help Membership Inference Attacks?cited by 2
  10. 2025Privacy-Utility-Bias Trade-offs for Privacy-Preserving Recommender Systemscited by 2
  11. 2024Exposing the Vulnerability of Decentralized Learning to Membership Inference Attacks Through the Lens of Graph Mixingcited by 1
  12. 2025SoK: Can Synthetic Images Replace Real Data? A Survey of Utility and Privacy of Synthetic Image Generationcited by 1
  13. 2026DeepLeak: Privacy Enhancing Hardening of Model Explanations Against Membership Leakagecited by 1
  14. 2026On Reliability of Membership Inference Vulnerability Evaluationcited by 1
  15. 2026What Does It Mean to Break a Distillation Defense?cited by 1
  16. 2024Adversarial Sample-Based Approach for Tighter Privacy Auditing in Final Model-Only Scenarioscited by 0
  17. 2025Towards the Anonymization of the Language Modelingcited by 0
  18. 2026SoK: Enhancing Cryptographic Collaborative Learning with Differential Privacycited by 0
  19. 2026Gaming the Metric, Not the Harm: Certifying Safety Audits against Strategic Platform Manipulationcited by 0
  20. 2026Performative Privacy: When Differential Privacy Maximizes Utilitycited by 0
  21. 2026The Privacy-Hallucination Tradeoff in Differentially Private Language Modelscited by 0
  22. 2026Calibrating One-Round Membership Inference with Neighborscited by 0
  23. 2026A Sharp Transition in Data Reconstruction under Differential Privacycited by 0