mlsanityData to 7 Oct 2026

Counterfactual Memorization in Neural Language Models

arXiv 2112.12938 ↗First posted 24 Dec 2021cs.CL cs.AI cs.LGcited by 155

Not on the map. Why?

Cited by 28 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Modern neural language models that are widely used in various NLP tasks risk memorizing sensitive information from their training data. Understanding this memorization is important in real world applications and also from a learning-theoretical perspective. An open question in previous studies of language model memorization is how to filter out "common" memorization. In fact, most memorization criteria strongly correlate with the number of occurrences in the training set, capturing memorized familiar phrases, public knowledge, templated texts, or other repeated data. We formulate a notion of counterfactual memorization which characterizes how a model's predictions change if a particular document is omitted during training. We identify and study counterfactually-memorized training examples in standard text datasets. We estimate the influence of each memorized training example on the validation set and on generated texts, showing how this can provide direct evidence of the source of memorization at test time.

Key facts. Cited by 155 papers in the map (data through 7 Oct 2026). Builds on 25 of its 61 references that are in the map. First posted on arXiv 24 Dec 2021.

Builds on

25 of 61 references.

References · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  8. 2020Extracting Training Data from Large Language Modelscited by 2,144
  9. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  10. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  11. 2021Membership Inference Attacks From First Principlescited by 839
  12. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
13 more
  1. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 565
  2. 2019Defending Against Neural Fake Newscited by 553
  3. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 492
  4. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 452
  5. 2019White-box vs Black-box: Bayes Optimal Strategies for Membership Inferencecited by 265
  6. 2020Influence Functions in Deep Learning Are Fragilecited by 247
  7. 2020Auditing Differentially Private Machine Learning: How Private is Private SGD?cited by 213
  8. 2021Adversary Instantiation: Lower Bounds for Differentially Private Machine Learningcited by 193
  9. 2021The Curious Case of Hallucinations in Neural Machine Translationcited by 161
  10. 2019On the Accuracy of Influence Functions for Measuring Group Effectscited by 146
  11. 2019Analyzing Information Leakage of Updates to Natural Language Modelscited by 89
  12. 2017Ethical Challenges in Data-Driven Dialogue Systemscited by 86
  13. 2020Understanding Unintended Memorization in Federated Learningcited by 36

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 17; 2023: 37; 2024: 44; 2025: 40; 2026: 17.

155 papers.

Sort:

By area:

  1. 2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
  2. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  3. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  4. 2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
  5. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  6. 2023Detecting Pretraining Data from Large Language Modelscited by 375
  7. 2023Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflictscited by 333
  8. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 309
  9. 2023Membership Inference Attacks against Language Models via Neighbourhood Comparisoncited by 249
  10. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 218
  11. 2024The Good and The Bad: Exploring Privacy Issues in Retrieval-Augmented Generation (RAG)cited by 159
  12. 2023The BigScience ROOTS Corpus: A 1.6TB Composite Multilingual Datasetcited by 148
38 more
  1. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoningcited by 142
  2. 2022Training Data Influence Analysis and Estimation: A Surveycited by 134
  3. 2022The Privacy Onion Effect: Memorization is Relativecited by 128
  4. 2023Speak, Memory: An Archaeology of Books Known to ChatGPT/GPT-4cited by 123
  5. 2023Foundation Models and Fair Usecited by 112
  6. 2023Low-Cost High-Power Membership Inference Attackscited by 99
  7. 2022Preventing Verbatim Memorization in Language Models Gives a False Sense of Privacycited by 97
  8. 2023Practical Membership Inference Attacks against Fine-tuned Large Language Models via Self-prompt Calibrationcited by 90
  9. 2025MATH-Perturb: Benchmarking LLMs' Math Reasoning Abilities against Hard Perturbationscited by 88
  10. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 86
  11. 2024Generalization v.s. Memorization: Tracing Language Models' Capabilities Back to Pretraining Datacited by 83
  12. 2023Flocks of Stochastic Parrots: Differentially Private Prompt Learning for Large Language Modelscited by 80
  13. 2024Follow My Instruction and Spill the Beans: Scalable Data Extraction from Retrieval-Augmented Generation Systemscited by 76
  14. 2022Do Language Models Plagiarize?cited by 72
  15. 2024DE-COP: Detecting Copyrighted Content in Language Models Training Datacited by 72
  16. 2023SILO Language Models: Isolating Legal Risk In a Nonparametric Datastorecited by 71
  17. 2023On the Generalization Properties of Diffusion Modelscited by 67
  18. 2022Recitation-Augmented Language Modelscited by 65
  19. 2022Understanding Transformer Memorization Recall Through Idiomscited by 62
  20. 2023"It's a Fair Game", or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational Agentscited by 60
  21. 2022Measuring Causal Effects of Data Statistics on Language Model's `Factual' Predictionscited by 57
  22. 2025How much do language models memorize?cited by 53
  23. 2022Measuring the Effect of Training Data on Deep Learning Predictions via Randomized Experimentscited by 50
  24. 2023Privacy in Large Language Models: Attacks, Defenses and Future Directionscited by 48
  25. 2024Towards Understanding the Working Mechanism of Text-to-Image Diffusion Modelcited by 48
  26. 2023Exploring Memorization in Fine-tuned Language Modelscited by 44
  27. 2022Memorization in NLP Fine-tuning Methodscited by 43
  28. 2023Training Data Extraction From Pre-trained Language Models: A Surveycited by 43
  29. 2024To Each (Textual Sequence) Its Own: Improving Memorized-Data Unlearning in Large Language Modelscited by 42
  30. 2024Evaluating Copyright Takedown Methods for Language Modelscited by 42
  31. 2023Controlling the Extraction of Memorized Data from Large Language Models via Prompt-Tuningcited by 40
  32. 2023Data Contamination Through the Lens of Timecited by 39
  33. 2024Proving membership in LLM pretraining data via data watermarkscited by 38
  34. 2024LLM-PBE: Assessing Data Privacy in Large Language Modelscited by 38
  35. 2023User Inference Attacks on Large Language Modelscited by 36
  36. 2024Copyright Protection in Generative AI: A Technical Perspectivecited by 35
  37. 2022Differentially Private Decoding in Large Language Modelscited by 34
  38. 2024Training Data Attribution via Approximate Unrolled Differentiationcited by 33

50 most cited of 155 All 155 (JSON)