mlsanityData to 7 Oct 2026

Knowledge Unlearning for Mitigating Privacy Risks in Language Models

arXiv 2210.01504 ↗First posted 4 Oct 2022cs.CLcited by 424

On the map: LLM safety and security → Machine unlearning

Abstract (arXiv)

Pretrained Language Models (LMs) memorize a vast amount of knowledge during initial pretraining, including information that may violate the privacy of personal lives and identities. Previous work addressing privacy issues for language models has mostly focused on data preprocessing and differential privacy methods, both requiring re-training the underlying LM. We propose knowledge unlearning as an alternative method to reduce privacy risks for LMs post hoc. We show that simply performing gradient ascent on target token sequences is effective at forgetting them with little to no degradation of general language modeling performances for larger LMs; it sometimes even substantially improves the underlying LM with just a few iterations. We also find that sequential unlearning is better than trying to unlearn all the data at once and that unlearning is highly dependent on which kind of data (domain) is forgotten. By showing comparisons with a previous data preprocessing method and a decoding method known to mitigate privacy risks for LMs, we show that unlearning can give a stronger empirical privacy guarantee in scenarios where the data vulnerable to extraction attacks are known a priori while being much more efficient and robust. We release the code and dataset needed to replicate our results at https://github.com/joeljang/knowledge-unlearning.

Key facts. Cited by 424 papers in the map (data through 7 Oct 2026). Placed in Machine unlearning, one of 12 directions in LLM safety and security. Builds on 42 of its 55 references that are in the map. First posted on arXiv 4 Oct 2022.

Builds on

42 of 55 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Deep Unlearning via Randomized Conditionally Independent Hessians15 Apr

Other references · 41

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  4. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  6. 2016Deep Learning with Differential Privacycited by 2,682
  7. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  8. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  9. 2020Extracting Training Data from Large Language Modelscited by 2,144
  10. 2019Language Models as Knowledge Bases?cited by 1,922
  11. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  12. 2019Machine Unlearningcited by 1,020
29 more
  1. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  2. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  3. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  4. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  5. 2019Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networkscited by 599
  6. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  7. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  8. 2019Making AI Forget You: Data Deletion in Machine Learningcited by 447
  9. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  10. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  11. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 309
  12. 2020Amnesiac Machine Learningcited by 305
  13. 2021Mind the Gap: Assessing Temporal Generalization in Neural Language Modelscited by 226
  14. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 218
  15. 2021Internet-Augmented Dialogue Generationcited by 217
  16. 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 195
  17. 2019WinoGrande: An Adversarial Winograd Schema Challenge at Scalecited by 194
  18. 2022What Does it Mean for a Language Model to Preserve Privacy?cited by 194
  19. 2021Towards Continual Knowledge Learning of Language Modelscited by 172
  20. 2020Can You Put it All Together: Evaluating Conversational Agents' Ability to Blend Skillscited by 130
  21. 2022The Privacy Onion Effect: Memorization is Relativecited by 128
  22. 2022Measuring Forgetting of Memorized Training Examplescited by 118
  23. 2021Large-Scale Differentially Private BERTcited by 117
  24. 2022TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Modelscited by 117
  25. 2021Improving and Simplifying Pattern Exploiting Trainingcited by 97
  26. 2021Does BERT Pretrained on Clinical Notes Reveal Sensitive Data?cited by 91
  27. 2016De-identification of Patient Notes with Recurrent Neural Networkscited by 88
  28. 2022Fortuitous Forgetting in Connectionist Networkscited by 39
  29. 2022Differentially Private Decoding in Large Language Modelscited by 34

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 16; 2024: 122; 2025: 142; 2026: 141.

424 papers.

Sort:

By area:

  1. 2024TOFU: A Task of Fictitious Unlearning for LLMscited by 507
  2. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 488
  3. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearningcited by 484
  4. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  5. 2023Who's Harry Potter? Approximate Unlearning in LLMscited by 366
  6. 2024MUSE: Machine Unlearning Six-Way Evaluation for Language Modelscited by 251
  7. 2024Rethinking Machine Unlearning for Large Language Modelscited by 247
  8. 2023In-Context Unlearning: Language Models as Few Shot Unlearnerscited by 214
  9. 2023Unlearn What You Want to Forget: Efficient Unlearning for LLMscited by 211
  10. 2024Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearningcited by 160
  11. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
  12. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
38 more
  1. 2024Eight Methods to Evaluate Robust Unlearning in LLMscited by 153
  2. 2024Machine Unlearning of Pre-trained Large Language Modelscited by 118
  3. 2024An Adversarial Perspective on Machine Unlearning for AI Safetycited by 113
  4. 2024Large Language Model Unlearning via Embedding-Corrupted Promptscited by 112
  5. 2024SOUL: Unlocking the Power of Second-Order Optimization for LLM Unlearningcited by 104
  6. 2024Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Differencecited by 85
  7. 2022Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Surveycited by 80
  8. 2024Inexact Unlearning Needs More Careful Evaluations to Avoid a False Sense of Privacycited by 78
  9. 2024RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Modelscited by 77
  10. 2024Catastrophic Failure of LLM Unlearning via Quantizationcited by 76
  11. 2025Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyondcited by 76
  12. 2024Reliable, Adaptable, and Attributable Language Models with Retrievalcited by 73
  13. 2023SILO Language Models: Isolating Legal Risk In a Nonparametric Datastorecited by 71
  14. 2024LLM Unlearning via Loss Adjustment with Only Forget Datacited by 71
  15. 2024Do Unlearning Methods Remove Information from Language Model Weights?cited by 66
  16. 2023"It's a Fair Game", or Is It? Examining How Users Navigate Disclosure Risks and Benefits When Using LLM-Based Conversational Agentscited by 60
  17. 2024A Closer Look at Machine Unlearning for Large Language Modelscited by 54
  18. 2024On Large Language Model Continual Unlearningcited by 51
  19. 2024Be like a Goldfish, Don't Memorize! Mitigating Memorization in Generative LLMscited by 50
  20. 2024UNDIAL: Self-Distillation with Adjusted Logits for Robust Unlearning in Large Language Modelscited by 49
  21. 2024Machine Unlearning in Generative AI: A Surveycited by 49
  22. 2024Eraser: Jailbreaking Defense in Large Language Models via Unlearning Harmful Knowledgecited by 48
  23. 2024UnlearnCanvas: Stylized Image Dataset for Enhanced Machine Unlearning Evaluation in Diffusion Modelscited by 47
  24. 2023Can Language Models be Instructed to Protect Personal Information?cited by 46
  25. 2025Measuring Chain of Thought Faithfulness by Unlearning Reasoning Stepscited by 43
  26. 2024To Each (Textual Sequence) Its Own: Improving Memorized-Data Unlearning in Large Language Modelscited by 42
  27. 2024Soft Prompting for Unlearning in Large Language Modelscited by 42
  28. 2025Unlearning Isn't Deletion: Investigating Reversibility of Machine Unlearning in LLMscited by 41
  29. 2025Exploring Criteria of Loss Reweighting to Enhance LLM Unlearningcited by 39
  30. 2024Digital Forgetting in Large Language Models: A Survey of Unlearning Methodscited by 38
  31. 2024LLM-PBE: Assessing Data Privacy in Large Language Modelscited by 38
  32. 2025LLM Unlearning via Neural Activation Redirectioncited by 38
  33. 2023Knowledge Unlearning for LLMs: Tasks, Methods, and Challengescited by 37
  34. 2024Selective Forgetting: Advancing Machine Unlearning Techniques and Evaluation in Language Modelscited by 36
  35. 2024Towards Robust and Parameter-Efficient Knowledge Unlearning for LLMscited by 36
  36. 2024CLEAR: Character Unlearning in Textual and Visual Modalitiescited by 36
  37. 2022Privacy Adhering Machine Un-learning in NLPcited by 35
  38. 2024Machine Unlearning Fails to Remove Data Poisoning Attackscited by 35

50 most cited of 424 All 424 (JSON)