mlsanityData to 14 Oct 2026

Privacy Ripple Effects from Adding or Removing Personal Information in Language Model Training

arXiv 2502.15680 ↗First posted 21 Feb 2025cs.CL cs.CRcited by 10

Not on the map. Why?

Abstract (arXiv)

Due to the sensitive nature of personally identifiable information (PII), its owners may have the authority to control its inclusion or request its removal from large-language model (LLM) training. Beyond this, PII may be added or removed from training datasets due to evolving dataset curation techniques, because they were newly scraped for retraining, or because they were included in a new downstream fine-tuning stage. We find that the amount and ease of PII memorization is a dynamic property of a model that evolves throughout training pipelines and depends on commonly altered design choices. We characterize three such novel phenomena: (1) similar-appearing PII seen later in training can elicit memorization of earlier-seen sequences in what we call assisted memorization, and this is a significant factor (in our settings, up to 1/3); (2) adding PII can increase memorization of other PII significantly (in our settings, as much as ≈!7.5×); and (3) removing PII can lead to other PII being memorized. Model creators should consider these first- and second-order privacy risks when training models to avoid the risk of new PII regurgitation.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Builds on 48 of its 65 references that are in the map. First posted on arXiv 21 Feb 2025.

Builds on

48 of 65 references.

References · 48

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  4. 2016Pointer Sentinel Mixture Modelscited by 3,096
  5. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  6. 2020Learning to summarize from human feedbackcited by 2,588
  7. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  8. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  10. 2018Hierarchical Neural Story Generationcited by 1,193
  11. 2019Machine Unlearningcited by 1,061
  12. 2021Membership Inference Attacks From First Principlescited by 878
36 more
  1. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
  2. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  3. 2018ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Modelscited by 498
  4. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 434
  5. 2020Label-Only Membership Inference Attackscited by 388
  6. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 314
  7. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 285
  8. 2019White-box vs Black-box: Bayes Optimal Strategies for Membership Inferencecited by 271
  9. 2020Approximate Data Deletion from Machine Learning Modelscited by 255
  10. 2019High Accuracy and High Fidelity Extraction of Neural Networkscited by 214
  11. 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 200
  12. 2020When Machine Unlearning Jeopardizes Privacycited by 198
  13. 2023Continual Pre-training of Language Modelscited by 198
  14. 2023Progressive Prompts: Continual Learning for Language Modelscited by 197
  15. 2023MADLAD-400: A Multilingual And Document-Level Large Audited Datasetcited by 181
  16. 2021Towards Continual Knowledge Learning of Language Modelscited by 174
  17. 2021On the Necessity of Auditable Algorithmic Definitions for Machine Unlearningcited by 171
  18. 2024CodeGemma: Open Code Models Based on Gemmacited by 150
  19. 2023Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4cited by 125
  20. 2022Measuring Forgetting of Memorized Training Examplescited by 118
  21. 2021Lifelong Pretraining: Continually Adapting Language Models to Emerging Corporacited by 115
  22. 2022Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Datasetcited by 113
  23. 2022Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacycited by 100
  24. 2024Rethinking LLM Memorization through the Lens of Adversarial Compressioncited by 94
  25. 2021Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?cited by 93
  26. 2024Inexact Unlearning Needs More Careful Evaluations to Avoid a False Sense of Privacycited by 82
  27. 2024DE-COP: Detecting Copyrighted Content in Language Models Training Datacited by 74
  28. 2024Demystifying Verbatim Memorization in Large Language Modelscited by 56
  29. 2023Exploring Memorization in Fine-tuned Language Modelscited by 45
  30. 2023Controlling the Extraction of Memorized Data from Large Language Models via Prompt-Tuningcited by 40
  31. 2023Students Parrot Their Teachers: Membership Inference on Model Distillationcited by 33
  32. 2024Teach LLMs to Phish: Stealing Private Information from Language Modelscited by 32
  33. 2024Recite, Reconstruct, Recollect: Memorization in LMs as a Multifaceted Phenomenoncited by 31
  34. 2024Unlocking Memorization in Large Language Models with Dynamic Soft Promptingcited by 21
  35. 2023What can we learn from Data Leakage and Unlearning for Law?cited by 15
  36. 2024Uncovering Latent Memories: Assessing Data Leakage and Memorization Patterns in Frontier AI Modelscited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 7.

10 papers.

  1. 2025Position: Privacy Is Not Just Memorization!cited by 17
  2. 2026Alignment Whack-a-Mole : Finetuning Activates Verbatim Recall of Copyrighted Books in Large Language Modelscited by 6
  3. 2025PATCH: Mitigating PII Leakage in Language Models with Privacy-Aware Targeted Circuit PatcHingcited by 2
  4. 2026Memorization Dynamics in Knowledge Distillation for Language Modelscited by 2
  5. 2026LACUNA: A Testbed for Evaluating Localization Precision for LLM Unlearningcited by 2
  6. 2025Randomized Masked Finetuning: An Efficient Way to Mitigate Memorization of PIIs in LLMscited by 0
  7. 2026Personal Information Parroting in Language Modelscited by 0
  8. 2026Reconstruction of Personally Identifiable Information from Proprietary Data in Supervised Fine-Tuned Modelscited by 0
  9. 2026Probe Choice Changes Canary-Memorization Verdicts: Three Post-Hoc Disagreement Case Studies in a Text-Dominant LoRA-Tuned Autoregressive Testbedcited by 0
  10. 2026Mitigating Private Data Leakage in LLMs with Whiteoutcited by 0