mlsanityData to 7 Oct 2026

Measuring Forgetting of Memorized Training Examples

arXiv 2207.00099 ↗First posted 30 Jun 2022cs.LGcited by 118

Not on the map. Why?

Cited by 29 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Machine learning models exhibit two seemingly contradictory phenomena: training data memorization, and various forms of forgetting. In memorization, models overfit specific training examples and become susceptible to privacy attacks. In forgetting, examples which appeared early in training are forgotten by the end. In this work, we connect these phenomena. We propose a technique to measure to what extent models "forget" the specifics of training examples, becoming less susceptible to privacy attacks on examples they have not seen recently. We show that, while non-convex models can memorize data forever in the worst-case, standard image, speech, and language models empirically do forget examples over time. We identify nondeterminism as a potential explanation, showing that deterministically trained models do not forget. Our results suggest that examples seen early when training with extremely large datasets - for instance those examples used to pre-train a model - may observe privacy benefits at the expense of examples seen later.

Key facts. Cited by 118 papers in the map (data through 7 Oct 2026). Builds on 66 of its 90 references that are in the map. First posted on arXiv 30 Jun 2022.

Builds on

66 of 90 references.

References · 66

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  10. 2016Deep Learning with Differential Privacycited by 2,682
  11. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  12. 2020Extracting Training Data from Large Language Modelscited by 2,144
38 more
  1. 2020Conformer: Convolution-augmented Transformer for Speech Recognitioncited by 1,838
  2. 2019Machine Unlearningcited by 1,020
  3. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  4. 2021Membership Inference Attacks From First Principlescited by 839
  5. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  6. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  7. 2017Renyi Differential Privacycited by 643
  8. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  9. 2019Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networkscited by 599
  10. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  11. 2018Do Better ImageNet Models Transfer Better?cited by 560
  12. 2019Certified Data Removal from Machine Learning Modelscited by 512
  13. 2019Making AI Forget You: Data Deletion in Machine Learningcited by 447
  14. 2021Remember What You Want to Forget: Algorithms for Machine Unlearningcited by 360
  15. 2017Measuring Catastrophic Forgetting in Neural Networkscited by 359
  16. 2016Semi-supervised Knowledge Transfer for Deep Learning from Private Training Datacited by 347
  17. 2021W2v-BERT: Combining Contrastive Learning and Masked Language Modeling for Self-Supervised Speech Pre-Trainingcited by 340
  18. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 309
  19. 2020Amnesiac Machine Learningcited by 305
  20. 2020Descent-to-Delete: Gradient-Based Methods for Machine Unlearningcited by 303
  21. 2013The Composition Theorem for Differential Privacycited by 290
  22. 2018Scalable Private Learning with PATEcited by 266
  23. 2019White-box vs Black-box: Bayes Optimal Strategies for Membership Inferencecited by 265
  24. 2020Approximate Data Deletion from Machine Learning Modelscited by 250
  25. 2021Enhanced Membership Inference Attacks against Machine Learning Modelscited by 237
  26. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 218
  27. 2020Auditing Differentially Private Machine Learning: How Private is Private SGD?cited by 213
  28. 2021Adversary Instantiation: Lower Bounds for Differentially Private Machine Learningcited by 193
  29. 2018Machine Learning with Membership Privacy using Adversarial Regularizationcited by 177
  30. 2020Mixed-Privacy Forgetting in Deep Networkscited by 158
  31. 2021On the Importance of Difficulty Calibration in Membership Inference Attackscited by 137
  32. 2013Hacking Smart Machines with Smarter Ones: How to Extract Meaningful Data from Machine Learning Classifierscited by 128
  33. 2021Machine Unlearning via Algorithmic Stabilitycited by 128
  34. 2019Updates-Leak: Data Set Inference and Reconstruction Attacks in Online Learningcited by 117
  35. 2020Variational Bayesian Unlearningcited by 114
  36. 2021Deep Learning with Label Differential Privacycited by 113
  37. 2018Privately Learning High-Dimensional Distributionscited by 103
  38. 2018Privacy Amplification by Iterationcited by 97

50 most cited of 66 All 66 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 34; 2024: 32; 2025: 30; 2026: 15.

118 papers.

Sort:

By area:

  1. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  2. 2023PaLM 2 Technical Reportcited by 1,154
  3. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  4. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Modelscited by 424
  5. 2022Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Modelscited by 418
  6. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  7. 2022A Survey of Machine Unlearningcited by 298
  8. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 278
  9. 2023Poisoning Language Models During Instruction Tuningcited by 254
  10. 2023In-Context Unlearning: Language Models as Few Shot Unlearnerscited by 214
  11. 2023Unlearn What You Want to Forget: Efficient Unlearning for LLMscited by 211
  12. 2023Understanding and Mitigating Copying in Diffusion Modelscited by 205
38 more
  1. 2024Do Membership Inference Attacks Work on Large Language Models?cited by 188
  2. 2023Emergent and Predictable Memorization in Large Language Modelscited by 147
  3. 2024Machine Unlearning of Pre-trained Large Language Modelscited by 118
  4. 2023Ambient Diffusion: Learning Clean Distributions from Corrupted Datacited by 108
  5. 2024What makes unlearning hard and what to do about itcited by 89
  6. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 86
  7. 2024Inexact Unlearning Needs More Careful Evaluations to Avoid a False Sense of Privacycited by 78
  8. 2023Bag of Tricks for Training Data Extraction from Language Modelscited by 67
  9. 2023On the Generalization Properties of Diffusion Modelscited by 67
  10. 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 66
  11. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 63
  12. 2023Computationally Budgeted Continual Learning: What Does Matter?cited by 61
  13. 2024Demystifying Verbatim Memorization in Large Language Modelscited by 55
  14. 2024Erasing Undesirable Influence in Diffusion Modelscited by 52
  15. 2023Training Data Extraction From Pre-trained Language Models: A Surveycited by 43
  16. 2024To Each (Textual Sequence) Its Own: Improving Memorized-Data Unlearning in Large Language Modelscited by 42
  17. 2022Canary in a Coalmine: Better Membership Inference with Ensembled Adversarial Queriescited by 36
  18. 2023User Inference Attacks on Large Language Modelscited by 36
  19. 2023SoK: Memorization in General-Purpose Large Language Modelscited by 36
  20. 2023Identifying and Mitigating Privacy Risks Stemming from Language Models: A Surveycited by 30
  21. 2024Teach LLMs to Phish: Stealing Private Information from Language Modelscited by 29
  22. 2023DiffInfinite: Large Mask-Image Synthesis via Parallel Random Patch Diffusion in Histopathologycited by 28
  23. 2023On The Impact of Machine Learning Randomness on Group Fairnesscited by 26
  24. 2024Learnable Privacy Neurons Localization in Language Modelscited by 26
  25. 2023MoPe: Model Perturbation-based Privacy Attacks on Language Modelscited by 25
  26. 2025Privacy Auditing of Large Language Modelscited by 22
  27. 2024Memorization in Self-Supervised Learning Improves Downstream Generalizationcited by 18
  28. 2024Adversarial Machine Unlearningcited by 16
  29. 2023Prefix Propagation: Parameter-Efficient Tuning for Long Sequencescited by 15
  30. 2023On the Trustworthiness Landscape of State-of-the-art Generative Models: A Survey and Outlookcited by 15
  31. 2024Memorization in deep learning: A surveycited by 14
  32. 2024How Much Can We Forget about Data Contamination?cited by 14
  33. 2023TMI! Finetuned Models Leak Private Information from their Pretraining Datacited by 13
  34. 2024Generalized Recorrupted-to-Recorrupted: Self-Supervised Learning Beyond Gaussian Noisecited by 13
  35. 2024Replication in Visual Diffusion Models: A Survey and Outlookcited by 11
  36. 2025Soup to go: mitigating forgetting during continual learning with model averagingcited by 11
  37. 2025Scalable Fingerprinting of Large Language Modelscited by 11
  38. 2025Blackbox Model Provenance via Palimpsestic Membership Inferencecited by 11

50 most cited of 118 All 118 (JSON)