mlsanityData to 7 Oct 2026

Quantifying Memorization Across Neural Language Models

arXiv 2202.07646 ↗First posted 15 Feb 2022cs.LG cs.CLcited by 755

Not on the map. Why?

Cited by 135 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Large language models (LMs) have been shown to memorize parts of their training data, and when prompted appropriately, they will emit the memorized training data verbatim. This is undesirable because memorization violates privacy (exposing user data), degrades utility (repeated easy-to-memorize text is often low quality), and hurts fairness (some texts are memorized over others). We describe three log-linear relationships that quantify the degree to which LMs emit memorized training data. Memorization significantly grows as we increase (1) the capacity of a model, (2) the number of times an example has been duplicated, and (3) the number of tokens of context used to prompt the model. Surprisingly, we find the situation becomes more complicated when generalizing these results across model families. On the whole, we find that memorization in LMs is more prevalent than previously believed and will likely get worse as models continues to scale, at least without active mitigations.

Key facts. Cited by 755 papers in the map (data through 7 Oct 2026). Builds on 21 of its 32 references that are in the map. First posted on arXiv 15 Feb 2022.

Builds on

21 of 32 references.

References · 21

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  4. 2016Deep Learning with Differential Privacycited by 2,682
  5. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  6. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  7. 2020Extracting Training Data from Large Language Modelscited by 2,144
  8. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  9. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  10. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  11. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 492
  12. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 309
9 more
  1. 2020Auditing Differentially Private Machine Learning: How Private is Private SGD?cited by 213
  2. 2022What Does it Mean for a Language Model to Preserve Privacy?cited by 194
  3. 2021Adversary Instantiation: Lower Bounds for Differentially Private Machine Learningcited by 193
  4. 2021Counterfactual Memorization in Neural Language Modelscited by 155
  5. 2021How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVENcited by 124
  6. 2021Large-Scale Differentially Private BERTcited by 117
  7. 2017Ethical Challenges in Data-Driven Dialogue Systemscited by 86
  8. 2020Training Production Language Models without Memorizing User Datacited by 68
  9. 2020Understanding Unintended Memorization in Federated Learningcited by 36

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 52; 2023: 168; 2024: 203; 2025: 178; 2026: 154.

755 papers.

Sort:

By area:

  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  2. 2023A Survey of Large Language Modelscited by 2,537
  3. 2022Emergent Abilities of Large Language Modelscited by 2,483
  4. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  5. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  6. 2023PaLM 2 Technical Reportcited by 1,154
  7. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  8. 2023BloombergGPT: A Large Language Model for Financecited by 923
  9. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  10. 2023Extracting Training Data from Diffusion Modelscited by 723
  11. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  12. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
38 more
  1. 2024Moshi: a speech-text foundation model for real-time dialoguecited by 536
  2. 2023Erasing Concepts from Diffusion Modelscited by 530
  3. 2023The Falcon Series of Open Language Modelscited by 526
  4. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearningcited by 484
  5. 2023MusicLM: Generating Music From Textcited by 442
  6. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 429
  7. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Modelscited by 424
  8. 2022Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Modelscited by 418
  9. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  10. 2023Scaling Data-Constrained Language Modelscited by 391
  11. 2023Causal Reasoning and Large Language Models: Opening a New Frontier for Causalitycited by 348
  12. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 333
  13. 2023Ablating Concepts in Text-to-Image Diffusion Modelscited by 312
  14. 2023Tool Learning with Foundation Modelscited by 302
  15. 2023Large Language Model Unlearningcited by 288
  16. 2022Draft, Sketch, and Prove: Guiding Formal Theorem Provers with Informal Proofscited by 278
  17. 2023Analyzing Leakage of Personally Identifiable Information in Language Modelscited by 278
  18. 2023TRAK: Attributing Model Behavior at Scalecited by 273
  19. 2022Prompting GPT-3 To Be Reliablecited by 258
  20. 2024RAFT: Adapting Language Model to Domain Specific RAGcited by 258
  21. 2024PoisonedRAG: Knowledge Corruption Attacks to Retrieval-Augmented Generation of Large Language Modelscited by 251
  22. 2023Membership Inference Attacks against Language Models via Neighbourhood Comparisoncited by 249
  23. 2024Rethinking Machine Unlearning for Large Language Modelscited by 247
  24. 2023Pretraining Language Models with Human Preferencescited by 245
  25. 2022Data Contamination: From Memorization to Exploitationcited by 234
  26. 2023Proving Test Set Contamination in Black Box Language Modelscited by 227
  27. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 221
  28. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 218
  29. 2023PromptRobust: Towards Evaluating the Robustness of Large Language Models on Adversarial Promptscited by 218
  30. 2023Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theorycited by 218
  31. 2023In-Context Unlearning: Language Models as Few Shot Unlearnerscited by 214
  32. 2024Repeat After Me: Transformers are Better than State Space Models at Copyingcited by 208
  33. 2023Understanding and Mitigating Copying in Diffusion Modelscited by 205
  34. 2023Inverse Scaling: When Bigger Isn't Bettercited by 205
  35. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 199
  36. 2022TALM: Tool Augmented Language Modelscited by 197
  37. 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 195
  38. 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 194

50 most cited of 755 All 755 (JSON)