mlsanityData to 7 Oct 2026

Unlearn What You Want to Forget: Efficient Unlearning for LLMs

arXiv 2310.20150 ↗First posted 31 Oct 2023cs.CL cs.AIcited by 211

On the map: LLM safety and security → Machine unlearning

Abstract (arXiv)

Large language models (LLMs) have achieved significant progress from pre-training on and memorizing a wide range of textual data, however, this process might suffer from privacy issues and violations of data protection regulations. As a result, the ability to easily remove data related to individual users from such models while not deteriorating their predictive quality after the removal becomes increasingly important. To address these issues, in this work, we propose an efficient unlearning framework that could efficiently update LLMs without having to retrain the whole model after data removals, by introducing lightweight unlearning layers learned with a selective teacher-student objective into the transformers. In addition, we introduce a fusion mechanism to effectively combine different unlearning layers that learns to forget different sets of data to handle a sequence of forgetting operations. Experiments on classification and generation tasks demonstrate the effectiveness of our proposed methods compared to the state-of-the-art baselines.

Key facts. Cited by 211 papers in the map (data through 7 Oct 2026). Placed in Machine unlearning, one of 12 directions in LLM safety and security. Builds on 38 of its 46 references that are in the map. First posted on arXiv 31 Oct 2023.

Builds on

38 of 46 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Privacy Adhering Machine Un-learning in NLP19 Dec
  2. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Models4 Oct
  3. 2021Remember What You Want to Forget: Algorithms for Machine Unlearning4 Mar

Other references · 35

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  10. 2016Deep Learning with Differential Privacycited by 2,682
  11. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  12. 2020Extracting Training Data from Large Language Modelscited by 2,144
23 more
  1. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  2. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  3. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  4. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  5. 2023LIMA: Less Is More for Alignmentcited by 1,080
  6. 2019Machine Unlearningcited by 1,020
  7. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  8. 2019Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networkscited by 599
  9. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  10. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  11. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  12. 2021Fast Model Editing at Scalecited by 491
  13. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  14. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  15. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  16. 2021Do Prompt-Based Models Really Understand the Meaning of their Prompts?cited by 324
  17. 2020K-Adapter: Infusing Knowledge into Pre-Trained Models with Adapterscited by 318
  18. 2020Approximate Data Deletion from Machine Learning Modelscited by 250
  19. 2017What do Neural Machine Translation Models Learn about Morphology?cited by 165
  20. 2021Entailment as Few-Shot Learnercited by 122
  21. 2022Measuring Forgetting of Memorized Training Examplescited by 118
  22. 2021Large-Scale Differentially Private BERTcited by 117
  23. 2023Parameter-Efficient Fine-Tuning Design Spacescited by 67

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 3; 2024: 101; 2025: 70; 2026: 37.

211 papers.

Sort:

By area:

  1. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearningcited by 484
  2. 2024Rethinking Machine Unlearning for Large Language Modelscited by 247
  3. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
  4. 2024A Comprehensive Study of Knowledge Editing for Large Language Modelscited by 157
  5. 2024Eight Methods to Evaluate Robust Unlearning in LLMscited by 153
  6. 2024Continual Learning for Large Language Models: A Surveycited by 144
  7. 2024Towards Safer Large Language Models through Machine Unlearningcited by 134
  8. 2023Receler: Reliable Concept Erasing of Text-to-Image Diffusion Models via Lightweight Eraserscited by 118
  9. 2024Machine Unlearning of Pre-trained Large Language Modelscited by 118
  10. 2024An Adversarial Perspective on Machine Unlearning for AI Safetycited by 113
  11. 2024Large Language Model Unlearning via Embedding-Corrupted Promptscited by 112
  12. 2024SOUL: Unlocking the Power of Second-Order Optimization for LLM Unlearningcited by 104
38 more
  1. 2024Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Differencecited by 85
  2. 2024Unlearning or Obfuscating? Jogging the Memory of Unlearned LLMs via Benign Relearningcited by 81
  3. 2024RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Modelscited by 77
  4. 2024Catastrophic Failure of LLM Unlearning via Quantizationcited by 76
  5. 2024LLM Unlearning via Loss Adjustment with Only Forget Datacited by 71
  6. 2024Knowledge Circuits in Pretrained Transformerscited by 60
  7. 2024Demystifying Verbatim Memorization in Large Language Modelscited by 55
  8. 2024Larimar: Large Language Models with Episodic Memory Controlcited by 53
  9. 2024On Large Language Model Continual Unlearningcited by 51
  10. 2024Direct Unlearning Optimization for Robust and Safe Text-to-Image Modelscited by 50
  11. 2024UNDIAL: Self-Distillation with Adjusted Logits for Robust Unlearning in Large Language Modelscited by 49
  12. 2024Machine Unlearning in Generative AI: A Surveycited by 49
  13. 2025LLM-based Agents Suffer from Hallucinations: A Survey of Taxonomy, Methods, and Directionscited by 49
  14. 2024Eraser: Jailbreaking Defense in Large Language Models via Unlearning Harmful Knowledgecited by 48
  15. 2024UnlearnCanvas: Stylized Image Dataset for Enhanced Machine Unlearning Evaluation in Diffusion Modelscited by 47
  16. 2024Single Image Unlearning: Efficient Machine Unlearning in Multimodal Large Language Modelscited by 44
  17. 2025Measuring Chain of Thought Faithfulness by Unlearning Reasoning Stepscited by 43
  18. 2024To Each (Textual Sequence) Its Own: Improving Memorized-Data Unlearning in Large Language Modelscited by 42
  19. 2024Soft Prompting for Unlearning in Large Language Modelscited by 42
  20. 2025Exploring Criteria of Loss Reweighting to Enhance LLM Unlearningcited by 39
  21. 2024Digital Forgetting in Large Language Models: A Survey of Unlearning Methodscited by 38
  22. 2025LLM Unlearning via Neural Activation Redirectioncited by 38
  23. 2023Knowledge Unlearning for LLMs: Tasks, Methods, and Challengescited by 37
  24. 2024Towards Robust and Parameter-Efficient Knowledge Unlearning for LLMscited by 36
  25. 2024CLEAR: Character Unlearning in Textual and Visual Modalitiescited by 36
  26. 2024Copyright Protection in Generative AI: A Technical Perspectivecited by 35
  27. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 35
  28. 2024Mechanistic Unlearning: Robust Knowledge Unlearning and Editing via Mechanistic Localizationcited by 34
  29. 2024Fast Exact Unlearning for In-Context Learning Data for LLMscited by 33
  30. 2024Offset Unlearning for Large Language Modelscited by 33
  31. 2024SHIELD: Evaluation and Defense Strategies for Copyright Compliance in LLM Text Generationcited by 33
  32. 2024Revisiting Who's Harry Potter: Towards Targeted Unlearning from a Causal Intervention Perspectivecited by 33
  33. 2024Alternate Preference Optimization for Unlearning Factual Knowledge in Large Language Modelscited by 32
  34. 2024PrE-Text: Training Language Models on Private Federated Data in the Age of LLMscited by 29
  35. 2024Undesirable Memorization in Large Language Models: A Surveycited by 29
  36. 2024Unlearning as multi-task optimization: A normalized gradient difference approach with an adaptive learning ratecited by 29
  37. 2024Intrinsic Test of Unlearning Using Parametric Knowledge Tracescited by 28
  38. 2024Corrective Machine Unlearningcited by 26

50 most cited of 211 All 211 (JSON)