mlsanityData to 7 Oct 2026

The Privacy Onion Effect: Memorization is Relative

arXiv 2206.10469 ↗First posted 21 Jun 2022cs.LG cs.CRcited by 128

Not on the map. Why?

Cited by 60 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Machine learning models trained on private datasets have been shown to leak their private data. While recent work has found that the average data point is rarely leaked, the outlier samples are frequently subject to memorization and, consequently, privacy leakage. We demonstrate and analyse an Onion Effect of memorization: removing the "layer" of outlier points that are most vulnerable to a privacy attack exposes a new layer of previously-safe points to the same attack. We perform several experiments to study this effect, and understand why it occurs. The existence of this effect has various consequences. For example, it suggests that proposals to defend against memorization without training with rigorous privacy guarantees are unlikely to be effective. Further, it suggests that privacy-enhancing technologies such as machine unlearning could actually harm the privacy of other users.

Key facts. Cited by 128 papers in the map (data through 7 Oct 2026). Builds on 28 of its 40 references that are in the map. First posted on arXiv 21 Jun 2022.

Builds on

28 of 40 references.

References · 28

  1. 2016Deep Learning with Differential Privacycited by 2,682
  2. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  3. 2020Extracting Training Data from Large Language Modelscited by 2,144
  4. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  5. 2021Membership Inference Attacks From First Principlescited by 839
  6. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 492
  7. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 452
  8. 2021Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarkscited by 388
  9. 2020Label-Only Membership Inference Attackscited by 371
  10. 2019Differential Privacy Has Disparate Impact on Model Accuracycited by 316
  11. 2020Amnesiac Machine Learningcited by 305
  12. 2021Unrolling SGD: Understanding Factors Influencing Machine Unlearningcited by 274
16 more
  1. 2019White-box vs Black-box: Bayes Optimal Strategies for Membership Inferencecited by 265
  2. 2020Differentially Private Learning Needs Better Features (or Much More Data)cited by 233
  3. 2020Auditing Differentially Private Machine Learning: How Private is Private SGD?cited by 213
  4. 2022Datamodels: Predicting Predictions from Training Datacited by 186
  5. 2019Stolen Memories: Leveraging Model Memorization for Calibrated White-Box Membership Inferencecited by 168
  6. 2021Counterfactual Memorization in Neural Language Modelscited by 155
  7. 2020When is Memorization of Irrelevant Training Data Necessary for High-Accuracy Learning?cited by 102
  8. 2022Truth Serum: Poisoning Machine Learning Models to Reveal Their Secretscited by 94
  9. 2020Characterizing Structural Regularities of Labeled Data in Overparameterized Modelscited by 93
  10. 2020Machine Unlearning: Linear Filtration for Logit-based Classifierscited by 89
  11. 2018Comprehensive Privacy Analysis of Deep Learning: Passive and Active White-box Inference Attacks against Centralized and Federated Learningcited by 75
  12. 2021EncoderMI: Membership Inference against Pre-trained Encoders in Contrastive Learningcited by 68
  13. 2020ML Privacy Meter: Aiding Regulatory Compliance by Quantifying the Privacy Risks of Machine Learningcited by 66
  14. 2017Knock Knock, Who's There? Membership Inference on Aggregate Location Datacited by 63
  15. 2021Membership Inference Attacks Against Recommender Systemscited by 52
  16. 2020Chasing Your Long Tails: Differentially Private Prediction in Health Care Settingscited by 37

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 6; 2023: 18; 2024: 44; 2025: 40; 2026: 20.

128 papers.

Sort:

By area:

  1. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Modelscited by 424
  2. 2023Unified Concept Editing in Diffusion Modelscited by 377
  3. 2024MACE: Mass Concept Erasure in Diffusion Modelscited by 298
  4. 2023Membership Inference Attacks against Language Models via Neighbourhood Comparisoncited by 249
  5. 2023Receler: Reliable Concept Erasing of Text-to-Image Diffusion Models via Lightweight Eraserscited by 118
  6. 2024Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Modelscited by 102
  7. 2024What makes unlearning hard and what to do about itcited by 89
  8. 2025MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbationscited by 88
  9. 2023Machine Unlearning: Solutions and Challengescited by 85
  10. 2024Inexact Unlearning Needs More Careful Evaluations to Avoid a False Sense of Privacycited by 78
  11. 2024Position: LLM Unlearning Benchmarks are Weak Measures of Progresscited by 58
  12. 2024Unveiling and Mitigating Memorization in Text-to-image Diffusion Models through Cross Attentioncited by 57
38 more
  1. 2024SoK: Membership Inference Attacks on LLMs are Rushing Nowhere (and How to Fix It)cited by 54
  2. 2024STEREO: A Two-Stage Framework for Adversarially Robust Concept Erasing from Text-to-Image Diffusion Modelscited by 39
  3. 2022Hidden Poison: Machine Unlearning Enables Camouflaged Poisoning Attackscited by 37
  4. 2022Canary in a Coalmine: Better Membership Inference with Ensembled Adversarial Queriescited by 36
  5. 2024Copyright Traps for Large Language Modelscited by 36
  6. 2024Finding NeMo: Localizing Neurons Responsible For Memorization in Diffusion Modelscited by 36
  7. 2024Evaluations of Machine Learning Privacy Defenses are Misleadingcited by 35
  8. 2024Machine Unlearning Fails to Remove Data Poisoning Attackscited by 35
  9. 2022Deep Regression Unlearningcited by 32
  10. 2023Students Parrot Their Teachers: Membership Inference on Model Distillationcited by 31
  11. 2024Recite, Reconstruct, Recollect: Memorization in LMs as a Multifaceted Phenomenoncited by 30
  12. 2023Achilles' Heels: Vulnerable Record Identification in Synthetic Data Publishingcited by 28
  13. 2023Traces of Memorisation in Large Language Models for Codecited by 26
  14. 2025CURE: Concept Unlearning via Orthogonal Representation Editing in Diffusion Modelscited by 24
  15. 2025Privacy Auditing of Large Language Modelscited by 22
  16. 2024Memorization in Self-Supervised Learning Improves Downstream Generalizationcited by 18
  17. 2025The Canary's Echo: Auditing Privacy Risks of LLM-Generated Synthetic Textcited by 18
  18. 2024Second-Order Information Matters: Revisiting Machine Unlearning for Large Language Modelscited by 17
  19. 2024AI Risk Management Should Incorporate Both Safety and Securitycited by 17
  20. 2024Semantic Membership Inference Attack against Large Language Modelscited by 17
  21. 2023What can we learn from Data Leakage and Unlearning for Law?cited by 15
  22. 2024Data Readiness for AI: A 360-Degree Surveycited by 15
  23. 2025SciHorizon: Benchmarking AI-for-Science Readiness from Scientific Data to Large Language Modelscited by 15
  24. 2023Understanding Reconstruction Attacks with the Neural Tangent Kernel and Dataset Distillationcited by 14
  25. 2024Memorization in deep learning: A surveycited by 14
  26. 2024Unstable Unlearning: The Hidden Risk of Concept Resurgence in Diffusion Modelscited by 14
  27. 2024Remaining-data-free Machine Unlearning by Suppressing Sample Contributioncited by 13
  28. 2024Memorized Images in Diffusion Models share a Subspace that can be Located and Deletedcited by 13
  29. 2022Data Isotopes for Data Provenance in DNNscited by 11
  30. 2023Gradients Look Alike: Sensitivity is Often Overestimated in DP-SGDcited by 11
  31. 2023MIST: Defending Against Membership Inference Attacks Through Membership-Invariant Subspace Trainingcited by 11
  32. 2024Trustworthy Federated Learning: Privacy, Security, and Beyondcited by 11
  33. 2023Ticketed Learning-Unlearning Schemescited by 10
  34. 2023Generation or Replication: Auscultating Audio Latent Diffusion Modelscited by 9
  35. 2024AI Data Readiness Inspector (AIDRIN) for Quantitative Assessment of Data Readiness for AIcited by 9
  36. 2024Localizing Memorization in SSL Vision Encoderscited by 9
  37. 2022How Does a Deep Learning Model Architecture Impact Its Privacy? A Comprehensive Study of Privacy Attacks on CNNs and Transformerscited by 8
  38. 2023Collaborative Learning via Prediction Consensuscited by 8

50 most cited of 128 All 128 (JSON)