mlsanityData to 14 Oct 2026

Recite, Reconstruct, Recollect: Memorization in LMs as a Multifaceted Phenomenon

arXiv 2406.17746 ↗First posted 25 Jun 2024cs.CL cs.AIcited by 31

Not on the map. Why?

Abstract (arXiv)

Memorization in language models is typically treated as a homogenous phenomenon, neglecting the specifics of the memorized data. We instead model memorization as the effect of a set of complex factors that describe each sample and relate it to the model and corpus. To build intuition around these factors, we break memorization down into a taxonomy: recitation of highly duplicated sequences, reconstruction of inherently predictable sequences, and recollection of sequences that are neither. We demonstrate the usefulness of our taxonomy by using it to construct a predictive model for memorization. By analyzing dependencies and inspecting the weights of the predictive model, we find that different factors influence the likelihood of memorization differently depending on the taxonomic category.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Builds on 26 of its 38 references that are in the map. First posted on arXiv 25 Jun 2024.

Builds on

26 of 38 references.

References · 26

  1. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  2. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  3. 2020Extracting Training Data from Large Language Modelscited by 2,189
  4. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  5. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,043
  6. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 864
  7. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  8. 2021Deduplicating Training Data Makes Language Models Bettercited by 714
  9. 2019Benign Overfitting in Linear Regressioncited by 549
  10. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 528
  11. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 460
  12. 2023Detecting Pretraining Data from Large Language Modelscited by 387
14 more
  1. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 314
  2. 2022What Does it Mean for a Language Model to Preserve Privacy?cited by 195
  3. 2021Counterfactual Memorization in Neural Language Modelscited by 156
  4. 2023Emergent and Predictable Memorization in Large Language Modelscited by 148
  5. 2022The Privacy Onion Effect: Memorization is Relativecited by 136
  6. 2023Copyright Violations and Large Language Modelscited by 108
  7. 2024Rethinking LLM Memorization through the Lens of Adversarial Compressioncited by 94
  8. 2024Copyright Traps for Large Language Modelscited by 37
  9. 2022Benign, Tempered, or Catastrophic: A Taxonomy of Overfittingcited by 36
  10. 2022Defending against Reconstruction Attacks with R'enyi Differential Privacycited by 32
  11. 2023Unveiling Memorization in Code Modelscited by 30
  12. 2022Provably Confidential Language Modellingcited by 13
  13. 2022Measures of Information Reflect Memorization Patternscited by 12
  14. 2023Memorisation Cartography: Mapping out the Memorisation-Generalisation Continuum in Neural Machine Translationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 9; 2025: 14; 2026: 8.

31 papers.

Sort:

By area:

  1. 2024On Memorization of Large Language Models in Logical Reasoningcited by 117
  2. 2024Demystifying Verbatim Memorization in Large Language Modelscited by 56
  3. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 51
  4. 2025Rethinking Memory in LLM based Agents: Representations, Operations, and Emerging Topicscited by 36
  5. 2024CopyBench: Measuring Literal and Non-Literal Reproduction of Copyright-Protected Text in Language Model Generationcited by 35
  6. 2024Undesirable Memorization in Large Language Models: A Surveycited by 29
  7. 2025VaultGemma: A Differentially Private Gemma Modelcited by 16
  8. 2025Analyzing Memorization in Large Language Models through the Lens of Model Attributioncited by 14
  9. 2024Beyond Math: Stories as a Testbed for Memorization-Constrained Reasoning in LLMscited by 12
  10. 2025Privacy Ripple Effects from Adding or Removing Personal Information in Language Model Trainingcited by 10
  11. 2025Extracting alignment data in open modelscited by 8
  12. 2024Composable Interventions for Language Modelscited by 4
19 more
  1. 2025OWL: Probing Cross-Lingual Recall of Memorized Texts via World Literaturecited by 4
  2. 2025Empirical Privacy Variancecited by 3
  3. 2025KScope: A Framework for Characterizing the Knowledge Status of Language Modelscited by 3
  4. 2026Position: Don't Just "Fix it in Post": A Science of AI Must Study Training Dynamicscited by 3
  5. 2024MemHunter: Automated and Verifiable Memorization Detection at Dataset-scale in LLMscited by 2
  6. 2025Memorization Inheritance in Sequence-Level Knowledge Distillation for Neural Machine Translationcited by 2
  7. 2025Privacy Leaks by Adversaries: Adversarial Iterations for Membership Inference Attackcited by 2
  8. 2025Mirage of Mastery: Memorization Tricks LLMs into Artificially Inflated Self-Knowledgecited by 2
  9. 2026A Prior-Aware Metric for Efficiently Distinguishing Memorization from Generalization in Large Language Modelscited by 2
  10. 2026Erase Persona, Forget Lore: Benchmarking Multimodal Copyright Unlearning in Large Vision Language Modelscited by 2
  11. 2025Low-Perplexity LLM-Generated Sequences and Where To Find Themcited by 1
  12. 2025Data Compressibility Quantifies LLM Memorizationcited by 1
  13. 2026Diffusion Models Preferentially Memorize Prototypical Examples or: Why Does My Diffusion Model Love Slop?cited by 1
  14. 2024Exposing Assumptions in AI Benchmarks through Cognitive Modellingcited by 0
  15. 2025Beyond Frequency: The Role of Redundancy in Large Language Model Memorizationcited by 0
  16. 2026Phantoms and Disclosures: A Statistical Framework for Auditing Privacy in Synthetic Datacited by 0
  17. 2026Enhancing SLMs for Sustainable Code Optimization in Radio-Astronomycited by 0
  18. 2026Generalization and Memorization along the Learning Trajectory of Neural Language Models: A Geometric Account of Categorizationcited by 0
  19. 2026Don't Forget! Decomposing the Training Dynamics of Memorization in Language Modelscited by 0