mlsanityData to 14 Oct 2026

Localizing Paragraph Memorization in Language Models

arXiv 2403.19851 ↗First posted 28 Mar 2024cs.CL cs.CR cs.LGcited by 31

Not on the map. Why?

Abstract (arXiv)

Can we localize the weights and mechanisms used by a language model to memorize and recite entire paragraphs of its training data? In this paper, we show that while memorization is spread across multiple layers and model components, gradients of memorized paragraphs have a distinguishable spatial pattern, being larger in lower model layers than gradients of non-memorized examples. Moreover, the memorized examples can be unlearned by fine-tuning only the high-gradient weights. We localize a low-layer attention head that appears to be especially involved in paragraph memorization. This head is predominantly focusing its attention on distinctive, rare tokens that are least frequent in a corpus-level unigram distribution. Next, we study how localized memorization is across the tokens in the prefix by perturbing tokens and measuring the caused change in the decoding. A few distinctive tokens early in a prefix can often corrupt the entire continuation. Overall, memorized continuations are not only harder to unlearn, but also to corrupt than non-memorized ones.

Key facts. Cited by 31 papers in the map (data through 14 Oct 2026). Builds on 26 of its 33 references that are in the map. First posted on arXiv 28 Mar 2024.

Builds on

26 of 33 references.

References · 26

  1. 2017Attention Is All You Needcited by 60,361
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  5. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  7. 2020Extracting Training Data from Large Language Modelscited by 2,189
  8. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  9. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  10. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 629
  11. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 528
  12. 2020What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimationcited by 497
14 more
  1. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 434
  2. 2023Who's Harry Potter? Approximate Unlearning in LLMscited by 381
  3. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 359
  4. 2023Membership Inference Attacks against Language Models via Neighbourhood Comparisoncited by 258
  5. 2023Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Modelscited by 243
  6. 2021Counterfactual Memorization in Neural Language Modelscited by 156
  7. 2021How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVENcited by 124
  8. 2023Future Lens: Anticipating Subsequent Tokens from a Single Hidden Statecited by 121
  9. 2022Understanding Transformer Memorization Recall Through Idiomscited by 62
  10. 2023Challenges with unsupervised LLM knowledge discoverycited by 31
  11. 2023Do Localization Methods Actually Localize Memorized Data in LLMs? A Tale of Two Benchmarkscited by 29
  12. 2022An Empirical Study of Memorization in NLPcited by 21
  13. 2023Unsupervised Contrast-Consistent Ranking with Language Modelscited by 13
  14. 2023Generalizing Backpropagation for Gradient-Based Interpretabilitycited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 14; 2025: 13; 2026: 4.

31 papers.

Sort:

By area:

  1. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 85
  2. 2024RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Modelscited by 78
  3. 2024Position: LLM Unlearning Benchmarks are Weak Measures of Progresscited by 63
  4. 2024Large Language Model Safety: A Holistic Surveycited by 59
  5. 2024Demystifying Verbatim Memorization in Large Language Modelscited by 56
  6. 2024Intrinsic Test of Unlearning Using Parametric Knowledge Tracescited by 29
  7. 2024Undesirable Memorization in Large Language Models: A Surveycited by 29
  8. 2024To Forget or Not? Towards Practical Knowledge Unlearning for Large Language Modelscited by 26
  9. 2024Dissecting Fine-Tuning Unlearning in Large Language Modelscited by 26
  10. 2024Controllable Context Sensitivity and the Knob Behind Itcited by 23
  11. 2024Causal Estimation of Memorisation Profilescited by 20
  12. 2025Why and How LLMs Hallucinate: Connecting the Dots with Subsequence Associationscited by 15
19 more
  1. 2025The Landscape of Memorization in LLMs: Mechanisms, Measurement, and Mitigationcited by 10
  2. 2024A Multi-Perspective Analysis of Memorization in Large Language Modelscited by 8
  3. 2025Mechanistic Indicators of Understanding in Large Language Modelscited by 8
  4. 2025Extracting alignment data in open modelscited by 8
  5. 2025Memorization Sinks: Isolating Memorization during LLM Trainingcited by 7
  6. 2024MemControl: Mitigating Memorization in Diffusion Models via Automated Parameter Selectioncited by 4
  7. 2024BTBR: A Bayesian-Theory-Driven Probabilistic-Fuzzy Framework for Implicit Bias Removal in Large Language Modelscited by 4
  8. 2025OWL: Probing Cross-Lingual Recall of Memorized Texts via World Literaturecited by 4
  9. 2025Understanding Verbatim Memorization in LLMs Through Circuit Discoverycited by 4
  10. 2025A Survey on Unlearning in Large Language Modelscited by 4
  11. 2025From Memorization to Reasoning in the Spectrum of Loss Curvaturecited by 2
  12. 2025Guess or Recall? Training CNNs to Classify and Localize Memorization in LLMscited by 1
  13. 2025Localizing and Mitigating Memorization in Image Autoregressive Modelscited by 1
  14. 2025Time Course MechInterp: Analyzing the Evolution of Components and Knowledge in Large Language Modelscited by 0
  15. 2025Beyond Frequency: The Role of Redundancy in Large Language Model Memorizationcited by 0
  16. 2026Detecting Data Contamination in Large Language Modelscited by 0
  17. 2026Metaphor Is Not All Attention Needscited by 0
  18. 2026Output Vector Editing for Memorization Mitigation in Large Language Modelscited by 0
  19. 2026Where Grokking Happens: Distributed Utility and Fourier Recoding Without a Module Switchcited by 0