mlsanityData to 7 Oct 2026

The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks

arXiv 1802.08232 ↗First posted 22 Feb 2018cs.LG cs.AI cs.CRcited by 847

Not on the map. Why?

Cited by 190 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

This paper describes a testing methodology for quantitatively assessing the risk that rare or unique training-data sequences are unintentionally memorized by generative sequence models---a common type of machine-learning model. Because such models are sometimes trained on sensitive data (e.g., the text of users' private messages), this methodology can benefit privacy by allowing deep-learning practitioners to select means of training that minimize such memorization. In experiments, we show that unintended memorization is a persistent, hard-to-avoid issue that can have serious consequences. Specifically, for models trained without consideration of memorization, we describe new, efficient procedures that can extract unique, secret sequences, such as credit card numbers. We show that our testing strategy is a practical and easy-to-use first line of defense, e.g., by describing its application to quantitatively limit data exposure in Google's Smart Compose, a commercial text-completion neural network trained on millions of users' email messages.

Key facts. Cited by 847 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 Feb 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 104; 2022: 146; 2023: 142; 2024: 189; 2025: 152; 2026: 114.

847 papers.

Sort:

By area:

  1. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  2. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,209
  3. 2023PaLM 2 Technical Reportcited by 1,154
  4. 2022Red Teaming Language Models with Language Modelscited by 983
  5. 2021Ethical and social risks of harm from Language Modelscited by 957
  6. 2021Membership Inference Attacks From First Principlescited by 839
  7. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  8. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  9. 2023Extracting Training Data from Diffusion Modelscited by 723
  10. 2023Erasing Concepts from Diffusion Modelscited by 530
  11. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  12. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
38 more
  1. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  2. 2021Remember What You Want to Forget: Algorithms for Machine Unlearningcited by 360
  3. 2023Ablating Concepts in Text-to-Image Diffusion Modelscited by 312
  4. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 309
  5. 2021Membership Inference Attacks on Machine Learning: A Surveycited by 306
  6. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 278
  7. 2021Unrolling SGD: Understanding Factors Influencing Machine Unlearningcited by 274
  8. 2024RAFT: Adapting Language Model to Domain Specific RAGcited by 258
  9. 2024Rethinking Machine Unlearning for Large Language Modelscited by 247
  10. 2023Pretraining Language Models with Human Preferencescited by 245
  11. 2021Enhanced Membership Inference Attacks against Machine Learning Modelscited by 237
  12. 2022Data Contamination: From Memorization to Exploitationcited by 234
  13. 2023Proving Test Set Contamination in Black Box Language Modelscited by 227
  14. 2021Machine Unlearning of Features and Labelscited by 223
  15. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 218
  16. 2022What Does it Mean for a Language Model to Preserve Privacy?cited by 194
  17. 2021Adversary Instantiation: Lower Bounds for Differentially Private Machine Learningcited by 193
  18. 2022Can Bad Teaching Induce Forgetting? Unlearning in Deep Networks using an Incompetent Teachercited by 189
  19. 2024Do Membership Inference Attacks Work on Large Language Models?cited by 188
  20. 2023Fast Machine Unlearning Without Retraining Through Selective Synaptic Dampeningcited by 177
  21. 2022Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attackscited by 168
  22. 2023Beyond Memorization: Violating Privacy Via Inference with Large Language Modelscited by 167
  23. 2023Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attackscited by 156
  24. 2021Counterfactual Memorization in Neural Language Modelscited by 155
  25. 2023How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacycited by 150
  26. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  27. 2023Emergent and Predictable Memorization in Large Language Modelscited by 147
  28. 2022Reconstructing Training Data with Informed Adversariescited by 143
  29. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoningcited by 142
  30. 2021On the Importance of Difficulty Calibration in Membership Inference Attackscited by 137
  31. 2023Auditing large language models: a three-layered approachcited by 135
  32. 2022Reconstructing Training Data from Trained Neural Networkscited by 129
  33. 2022The Privacy Onion Effect: Memorization is Relativecited by 128
  34. 2021The Distributed Discrete Gaussian Mechanism for Federated Learning with Secure Aggregationcited by 127
  35. 2022Zero-Shot Machine Unlearningcited by 125
  36. 2021How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVENcited by 124
  37. 2022Measuring Forgetting of Memorized Training Examplescited by 118
  38. 2024Machine Unlearning of Pre-trained Large Language Modelscited by 118

50 most cited of 847 All 847 (JSON)