mlsanityData to 7 Oct 2026

Deduplicating Training Data Mitigates Privacy Risks in Language Models

arXiv 2202.06539 ↗First posted 14 Feb 2022cs.CR cs.CL cs.LGcited by 309

Not on the map. Why?

Cited by 58 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Past work has shown that large language models are susceptible to privacy attacks, where adversaries generate sequences from a trained model and detect which sequences are memorized from the training set. In this work, we show that the success of these attacks is largely due to duplication in commonly used web-scraped training sets. We first show that the rate at which language models regenerate training sequences is superlinearly related to a sequence's count in the training set. For instance, a sequence that is present 10 times in the training data is on average generated ~1000 times more often than a sequence that is present only once. We next show that existing methods for detecting memorized sequences have near-chance accuracy on non-duplicated training sequences. Finally, we find that after applying methods to deduplicate training data, language models are considerably more secure against these types of privacy attacks. Taken together, our results motivate an increased focus on deduplication in privacy-sensitive applications and a reevaluation of the practicality of existing privacy attacks.

Key facts. Cited by 309 papers in the map (data through 7 Oct 2026). Builds on 29 of its 41 references that are in the map. First posted on arXiv 14 Feb 2022.

Builds on

29 of 41 references.

References · 29

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  5. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  6. 2020Extracting Training Data from Large Language Modelscited by 2,144
  7. 2019Language Models as Knowledge Bases?cited by 1,922
  8. 2018Hierarchical Neural Story Generationcited by 1,182
  9. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  10. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  11. 2021Membership Inference Attacks From First Principlescited by 839
  12. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 713
17 more
  1. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  2. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 492
  3. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  4. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  5. 2020Information Leakage in Embedding Modelscited by 199
  6. 2018Do CIFAR-10 Classifiers Generalize to CIFAR-10?cited by 197
  7. 2021Counterfactual Memorization in Neural Language Modelscited by 155
  8. 2021On the Importance of Difficulty Calibration in Membership Inference Attackscited by 137
  9. 2021How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVENcited by 124
  10. 2020When is Memorization of Irrelevant Training Data Necessary for High-Accuracy Learning?cited by 102
  11. 2021Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?cited by 91
  12. 2018Understanding Membership Inferences on Well-Generalized Learning Modelscited by 83
  13. 2018Towards Robust and Privacy-preserving Text Representationscited by 75
  14. 2021On Memorization in Probabilistic Deep Generative Modelscited by 60
  15. 2018Privacy-preserving Neural Representations of Textcited by 54
  16. 2021Privacy Regularization: Joint Privacy-Utility Optimization in Language Modelscited by 29
  17. 2020Reflective Decoding: Beyond Unidirectional Generation with Off-the-Shelf Language Modelscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 25; 2023: 69; 2024: 100; 2025: 68; 2026: 47.

309 papers.

Sort:

By area:

  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2023A Survey of Large Language Modelscited by 2,537
  3. 2022Emergent Abilities of Large Language Modelscited by 2,483
  4. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,104
  5. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  6. 2023Extracting Training Data from Diffusion Modelscited by 723
  7. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  8. 2022InCoder: A Generative Model for Code Infilling and Synthesiscited by 492
  9. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  10. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Modelscited by 424
  11. 2023Scaling Data-Constrained Language Modelscited by 391
  12. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
38 more
  1. 2023Detecting Pretraining Data from Large Language Modelscited by 375
  2. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  3. 2023SemDeDup: Data-efficient learning at web-scale through semantic deduplicationcited by 295
  4. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 278
  5. 2024RAFT: Adapting Language Model to Domain Specific RAGcited by 258
  6. 2023Poisoning Language Models During Instruction Tuningcited by 254
  7. 2023Membership Inference Attacks against Language Models via Neighbourhood Comparisoncited by 249
  8. 2022Data Contamination: From Memorization to Exploitationcited by 234
  9. 2023Proving Test Set Contamination in Black Box Language Modelscited by 227
  10. 2023Inverse Scaling: When Bigger Isn't Bettercited by 205
  11. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194
  12. 2024Do Membership Inference Attacks Work on Large Language Models?cited by 188
  13. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 179
  14. 2023Time Travel in LLMs: Tracing Data Contamination in Large Language Modelscited by 175
  15. 2022Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attackscited by 168
  16. 2024The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG)cited by 159
  17. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
  18. 2023Emergent and Predictable Memorization in Large Language Modelscited by 147
  19. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 138
  20. 2022Training Data Influence Analysis and Estimation: A Surveycited by 134
  21. 2024Rho-1: Not All Tokens Are What You Needcited by 132
  22. 2023Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4cited by 123
  23. 2022Measuring Forgetting of Memorized Training Examplescited by 118
  24. 2023Machine Learning for Synthetic Data Generation: A Reviewcited by 118
  25. 2022Pile of Law: Learning Responsible Data Filtering from the Law and a 256GB Open-Source Legal Datasetcited by 113
  26. 2023Foundation Models and Fair Usecited by 112
  27. 2023Skywork: A More Open Bilingual Foundation Modelcited by 110
  28. 2022Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipecited by 108
  29. 2022Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacycited by 97
  30. 2023On Provable Copyright Protection for Generative Modelscited by 97
  31. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  32. 2023Practical Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibrationcited by 90
  33. 2023Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicitycited by 86
  34. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 86
  35. 2022Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Surveycited by 80
  36. 2024Don't Listen To Me: Understanding and Exploring Jailbreak Prompts of Large Language Modelscited by 77
  37. 2023Use of LLMs for Illicit Purposes: Threats, Prevention Measures, and Vulnerabilitiescited by 76
  38. 2024Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systemscited by 76

50 most cited of 309 All 309 (JSON)