mlsanityData to 14 Oct 2026

Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacy

arXiv 2210.17546 ↗First posted 31 Oct 2022cs.LG cs.CLcited by 100

Not on the map. Why?

Cited by 21 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Studying data memorization in neural language models helps us understand the risks (e.g., to privacy or copyright) associated with models regurgitating training data and aids in the development of countermeasures. Many prior works -- and some recently deployed defenses -- focus on "verbatim memorization", defined as a model generation that exactly matches a substring from the training set. We argue that verbatim memorization definitions are too restrictive and fail to capture more subtle forms of memorization. Specifically, we design and implement an efficient defense that perfectly prevents all verbatim memorization. And yet, we demonstrate that this "perfect" filter does not prevent the leakage of training data. Indeed, it is easily circumvented by plausible and minimally modified "style-transfer" prompts -- and in some cases even the non-modified original prompts -- to extract memorized information. We conclude by discussing potential alternative definitions and why defining memorization is a difficult yet crucial open question for neural language models.

Key facts. Cited by 100 papers in the map (data through 14 Oct 2026). Builds on 30 of its 46 references that are in the map. First posted on arXiv 31 Oct 2022.

Builds on

30 of 46 references.

References · 30

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,512
  7. 2016Deep Learning with Differential Privacycited by 2,728
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  9. 2020Extracting Training Data from Large Language Modelscited by 2,189
  10. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
  11. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  12. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
18 more
  1. 2021Differentially Private Fine-tuning of Language Modelscited by 400
  2. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 314
  3. 2020ToTTo: A Controlled Table-To-Text Generation Datasetcited by 268
  4. 2019The Secret Revealer: Generative Model-Inversion Attacks Against Deep Neural Networkscited by 246
  5. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  6. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 220
  7. 2022What Does it Mean for a Language Model to Preserve Privacy?cited by 195
  8. 2021Counterfactual Memorization in Neural Language Modelscited by 156
  9. 2022Reconstructing Training Data with Informed Adversariescited by 148
  10. 2020DART: Open-Domain Structured Data Record to Text Generationcited by 145
  11. 2022Reconstructing Training Data from Trained Neural Networkscited by 133
  12. 2020MLSUM: The Multilingual Summarization Corpuscited by 121
  13. 2021Large-Scale Differentially Private BERTcited by 118
  14. 2020Training Production Language Models without Memorizing User Datacited by 68
  15. 2019Semantic Noise Matters for Neural Natural Language Generationcited by 64
  16. 2022Defending against Reconstruction Attacks with R'enyi Differential Privacycited by 32
  17. 2022Provably Confidential Language Modellingcited by 13
  18. 2017Bloom Filters, Adaptivity, and the Dictionary Problemcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 4; 2024: 39; 2025: 38; 2026: 19.

100 papers.

Sort:

By area:

  1. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 229
  2. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 147
  3. 2024AI Agents Under Threat: A Survey of Key Security Challenges and Future Pathwayscited by 127
  4. 2023Practical Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibrationcited by 98
  5. 2024Investigating Data Contamination for Pre-training Language Modelscited by 91
  6. 2023Effective Prompt Extraction from Language Modelscited by 85
  7. 2024On the Societal Impact of Open Foundation Modelscited by 65
  8. 2024Demystifying Verbatim Memorization in Large Language Modelscited by 56
  9. 2024A Closer Look at Machine Unlearning for Large Language Modelscited by 56
  10. 2025Measuring Chain of Thought Faithfulness by Unlearning Reasoning Stepscited by 47
  11. 2024Evaluating Copyright Takedown Methods for Language Modelscited by 43
  12. 2023User Inference Attacks on Large Language Modelscited by 36
38 more
  1. 2024SHIELD: Evaluation and Defense Strategies for Copyright Compliance in LLM Text Generationcited by 34
  2. 2024Alpaca against Vicuna: Using LLMs to Uncover Memorization of LLMscited by 33
  3. 2024Undesirable Memorization in Large Language Models: A Surveycited by 29
  4. 2024Evaluating n-Gram Novelty of Language Models Using Rusty-DAWGcited by 25
  5. 2025Landscape of Thoughts: Visualizing the Reasoning Process of Large Language Modelscited by 25
  6. 2024Avoiding Copyright Infringement via Large Language Model Unlearningcited by 24
  7. 2024CS4: Measuring the Creativity of Large Language Models Automatically by Controlling the Number of Story-Writing Constraintscited by 24
  8. 2024Prompt Leakage effect and defense strategies for multi-turn LLM interactionscited by 23
  9. 2025Privacy Auditing of Large Language Modelscited by 23
  10. 2024Unlocking Memorization in Large Language Models with Dynamic Soft Promptingcited by 21
  11. 2024Sampling-based Pseudo-Likelihood for Membership Inference Attackscited by 20
  12. 2024Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practicescited by 20
  13. 2025The Sum Leaks More Than Its Parts: Compositional Privacy Risks and Mitigations in Multi-Agent Collaborationcited by 20
  14. 2024Can Watermarking Large Language Models Prevent Copyrighted Text Generation and Hide Training Data?cited by 18
  15. 2024Reconstruct Your Previous Conversations! Comprehensively Investigating Privacy Leakage Risks in Conversations with GPT Modelscited by 16
  16. 2024Ward: Provable RAG Dataset Inference via LLM Watermarkscited by 16
  17. 2024Verifiable by Design: Aligning Language Models to Quote from Pre-Training Datacited by 13
  18. 2024Risks and Opportunities of Open-Source Generative AIcited by 13
  19. 2024REVS: Unlearning Sensitive Information in Language Models via Rank Editing in the Vocabulary Spacecited by 13
  20. 2025Blackbox Model Provenance via Palimpsestic Membership Inferencecited by 11
  21. 2024Not All Similarities Are Created Equal: Leveraging Data-Driven Biases to Inform GenAI Copyright Disputescited by 10
  22. 2024Disguised Copyright Infringement of Latent Diffusion Modelscited by 10
  23. 2024The Mosaic Memory of Large Language Modelscited by 10
  24. 2025Privacy Ripple Effects from Adding or Removing Personal Information in Language Model Trainingcited by 10
  25. 2025The Landscape of Memorization in LLMs: Mechanisms, Measurement, and Mitigationcited by 10
  26. 2025DCR: Quantifying Data Contamination in LLMs Evaluationcited by 10
  27. 2024An Exploratory Investigation into Code License Infringements in Large Language Model Training Datasetscited by 9
  28. 2024A Multi-Perspective Analysis of Memorization in Large Language Modelscited by 8
  29. 2024Towards More Realistic Extraction Attacks: An Adversarial Perspectivecited by 8
  30. 2025Information-Guided Identification of Training Data Imprint in (Proprietary) Large Language Modelscited by 8
  31. 2025Existing Large Language Model Unlearning Evaluations Are Inconclusivecited by 8
  32. 2024Copyright-Protected Language Generation via Adaptive Model Fusioncited by 7
  33. 2024The Unreasonable Ineffectiveness of Nucleus Sampling on Mitigating Text Memorizationcited by 6
  34. 2024Data Laundering: Artificially Boosting Benchmark Results through Knowledge Distillationcited by 6
  35. 2025The KL3M Data Project: Copyright-Clean Training Resources for Large Language Modelscited by 6
  36. 2025Beyond Data Privacy: New Privacy Risks for Large Language Modelscited by 6
  37. 2025SUV: Scalable Large Language Model Copyright Compliance with Regularized Selective Unlearningcited by 5
  38. 2024Strong Copyright Protection for Language Models via Adaptive Model Fusioncited by 4

50 most cited of 100 All 100 (JSON)