mlsanityData to 7 Oct 2026

Mitigating Memorization In Language Models

arXiv 2410.02159 ↗First posted 3 Oct 2024cs.LG cs.AI cs.CLcited by 11

On the map: LLM safety and security → Machine unlearning

Abstract (arXiv)

Language models (LMs) can "memorize" information, i.e., encode training data in their weights in such a way that inference-time queries can lead to verbatim regurgitation of that data. This ability to extract training data can be problematic, for example, when data are private or sensitive. In this work, we investigate methods to mitigate memorization: three regularizer-based, three finetuning-based, and eleven machine unlearning-based methods, with five of the latter being new methods that we introduce. We also introduce TinyMem, a suite of small, computationally-efficient LMs for the rapid development and evaluation of memorization-mitigation methods. We demonstrate that the mitigation methods that we develop using TinyMem can successfully be applied to production-grade LMs, and we determine via experiment that: regularizer-based mitigation methods are slow and ineffective at curbing memorization; fine-tuning-based methods are effective at curbing memorization, but overly expensive, especially for retaining higher accuracies; and unlearning-based methods are faster and more effective, allowing for the precise localization and removal of memorized information from LM weights prior to inference. We show, in particular, that our proposed unlearning method BalancedSubnet outperforms other mitigation methods at removing memorized information while preserving performance on target tasks.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Machine unlearning, one of 12 directions in LLM safety and security. Builds on 60 of its 77 references that are in the map. First posted on arXiv 3 Oct 2024.

Builds on

60 of 77 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024UnUnlearning: Unlearning is not sufficient for content regulation in advanced generative AI27 Jun
  2. 2024To Each (Textual Sequence) Its Own: Improving Memorized-Data Unlearning in Large Language Models6 May
  3. 2024Rethinking LLM Memorization through the Lens of Adversarial Compression23 Apr
  4. 2024Machine Unlearning of Pre-trained Large Language Models23 Feb
  5. 2024Rethinking Machine Unlearning for Large Language Models13 Feb
  6. 2023Who's Harry Potter? Approximate Unlearning in LLMs3 Oct

Other references · 54

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  3. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  5. 2016Pointer Sentinel Mixture Modelscited by 2,992
  6. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  8. 2020Extracting Training Data from Large Language Modelscited by 2,144
  9. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  10. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  11. 2019Machine Unlearningcited by 1,020
  12. 2017Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoningcited by 1,012
38 more
  1. 2022Mass-Editing Memory in a Transformercited by 976
  2. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  3. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  4. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  5. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  6. 2024A Survey on Large Language Models for Code Generationcited by 646
  7. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  8. 2021Editing Factual Knowledge in Language Modelscited by 548
  9. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  10. 2021Fast Model Editing at Scalecited by 491
  11. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 452
  12. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  13. 2018Adversarially Robust Generalization Requires More Datacited by 318
  14. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 309
  15. 2019What's Hidden in a Randomly Weighted Neural Network?cited by 236
  16. 2020Modifying Memories in Transformer Modelscited by 222
  17. 2018Implicit Self-Regularization in Deep Neural Networks: Evidence from Random Matrix Theory and Implications for Learningcited by 211
  18. 2023Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representationscited by 197
  19. 2017Spectral Norm Regularization for Improving the Generalizability of Deep Learningcited by 178
  20. 2020Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defensescited by 171
  21. 2023Beyond Memorization: Violating Privacy Via Inference with Large Language Modelscited by 167
  22. 2023Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attackscited by 156
  23. 2021Deep Learning Through the Lens of Example Difficultycited by 139
  24. 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
  25. 2020Predicting trends in the quality of state-of-the-art neural networks without access to training or testing datacited by 113
  26. 2023Copyright Violations and Large Language Modelscited by 108
  27. 2022Neurotoxin: Durable Backdoors in Federated Learningcited by 97
  28. 2020Improved Natural Language Generation via Loss Truncationcited by 75
  29. 2023DEPN: Detecting and Editing Privacy Neurons in Pretrained Language Modelscited by 75
  30. 2021On the geometry of generalization and memorization in deep neural networkscited by 71
  31. 2023Can Neural Network Memorization Be Localized?cited by 66
  32. 2023PURR: Efficiently Editing Language Model Hallucinations by Denoising Language Model Corruptionscited by 53
  33. 2024Be like a Goldfish, Don't Memorize! Mitigating Memorization in Generative LLMscited by 50
  34. 2023Rigorously Assessing Natural Language Explanations of Neuronscited by 44
  35. 2023How to estimate carbon footprint when training deep learning models? A guide and reviewcited by 34
  36. 2023Do Localization Methods Actually Localize Memorized Data in LLMs? A Tale of Two Benchmarkscited by 29
  37. 2024Teach LLMs to Phish: Stealing Private Information from Language Modelscited by 29
  38. 2021Mitigating Memorization of Noisy Labels via Regularization between Representationscited by 26

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 6.

11 papers.

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  2. 2026Attention Smoothing Is All You Need For Unlearningcited by 6
  3. 2026GSS: Gated Subspace Steering for Selective Memorization Mitigation in LLMscited by 2
  4. 2025A Lightweight Method to Disrupt Memorized Sequences in LLMcited by 1
  5. 2025From Memorization to Reasoning in the Spectrum of Loss Curvaturecited by 1
  6. 2026Landscaper: Understanding Loss Landscapes Through Multi-Dimensional Topological Analysiscited by 1
  7. 2025Topology-Aware Knowledge Propagation in Decentralized Learningcited by 0
  8. 2025Beyond Frequency: The Role of Redundancy in Large Language Model Memorizationcited by 0
  9. 2026Output Vector Editing for Memorization Mitigation in Large Language Modelscited by 0
  10. 2026Certification of Machine Learning Models via Directional Sharpnesscited by 0
  11. 2026Initialization Improves LLM-Driven Discoverycited by 0