mlsanityData to 7 Oct 2026

KGA: A General Machine Unlearning Framework Based on Knowledge Gap Alignment

arXiv 2305.06535 ↗First posted 11 May 2023cs.CLcited by 94

On the map: LLM safety and security → Machine unlearning

Abstract (arXiv)

Recent legislation of the "right to be forgotten" has led to the interest in machine unlearning, where the learned models are endowed with the function to forget information about specific training instances as if they have never existed in the training set. Previous work mainly focuses on computer vision scenarios and largely ignores the essentials of unlearning in NLP field, where text data contains more explicit and sensitive personal information than images. In this paper, we propose a general unlearning framework called KGA to induce forgetfulness. Different from previous work that tries to recover gradients or forces models to perform close to one specific distribution, KGA maintains distribution differences (i.e., knowledge gap). This relaxes the distribution assumption. Furthermore, we first apply the unlearning method to various NLP tasks (i.e., classification, translation, response generation) and propose several unlearning evaluation metrics with pertinence. Experiments on large-scale datasets show that KGA yields comprehensive improvements over baselines, where extensive analyses further validate the effectiveness of KGA and provide insight into unlearning for NLP tasks.

Key facts. Cited by 94 papers in the map (data through 7 Oct 2026). Placed in Machine unlearning, one of 12 directions in LLM safety and security. Builds on 16 of its 25 references that are in the map. First posted on arXiv 11 May 2023.

Builds on

16 of 25 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Can Bad Teaching Induce Forgetting? Unlearning in Deep Networks using an Incompetent Teacher17 May
  2. 2022Deep Unlearning via Randomized Conditionally Independent Hessians15 Apr

Other references · 14

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  5. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  6. 2019Machine Unlearningcited by 1,020
  7. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 820
  8. 2019Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networkscited by 599
  9. 2019Certified Data Removal from Machine Learning Modelscited by 512
  10. 2018ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Modelscited by 475
  11. 2019Making AI Forget You: Data Deletion in Machine Learningcited by 447
  12. 2020Forgetting Outside the Box: Scrubbing Deep Networks of Information Accessible from Input-Output Observationscited by 171
2 more
  1. 2021Challenges in Detoxifying Language Modelscited by 169
  2. 2021Knowledge-Adaptation Priorscited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 46; 2025: 29; 2026: 12.

94 papers.

Sort:

By area:

  1. 2024TOFU: A Task of Fictitious Unlearning for LLMscited by 507
  2. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearningcited by 484
  3. 2023Who's Harry Potter? Approximate Unlearning in LLMscited by 366
  4. 2024Rethinking Machine Unlearning for Large Language Modelscited by 247
  5. 2023In-Context Unlearning: Language Models as Few Shot Unlearnerscited by 214
  6. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
  7. 2024Eight Methods to Evaluate Robust Unlearning in LLMscited by 153
  8. 2024Large Language Model Unlearning via Embedding-Corrupted Promptscited by 112
  9. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  10. 2024SOUL: Unlocking the Power of Second-Order Optimization for LLM Unlearningcited by 104
  11. 2024LLM Unlearning via Loss Adjustment with Only Forget Datacited by 71
  12. 2024Machine Unlearning in Generative AI: A Surveycited by 49
38 more
  1. 2023Can Language Models be Instructed to Protect Personal Information?cited by 46
  2. 2024Single Image Unlearning: Efficient Machine Unlearning in Multimodal Large Language Modelscited by 44
  3. 2025Exploring Criteria of Loss Reweighting to Enhance LLM Unlearningcited by 39
  4. 2024Digital Forgetting in Large Language Models: A Survey of Unlearning Methodscited by 38
  5. 2024LLM-PBE: Assessing Data Privacy in Large Language Modelscited by 38
  6. 2023Knowledge Unlearning for LLMs: Tasks, Methods, and Challengescited by 37
  7. 2023SoK: Memorization in General-Purpose Large Language Modelscited by 36
  8. 2024Selective Forgetting: Advancing Machine Unlearning Techniques and Evaluation in Language Modelscited by 36
  9. 2024Towards Robust and Parameter-Efficient Knowledge Unlearning for LLMscited by 36
  10. 2025Rethinking Memory in LLM based Agents: Representations, Operations, and Emerging Topicscited by 36
  11. 2024Machine Unlearning Fails to Remove Data Poisoning Attackscited by 35
  12. 2024Offset Unlearning for Large Language Modelscited by 33
  13. 2024Revisiting Who's Harry Potter: Towards Targeted Unlearning from a Causal Intervention Perspectivecited by 33
  14. 2025SafeEraser: Enhancing Safety in Multimodal Large Language Models through Multimodal Machine Unlearningcited by 31
  15. 2025MMUnlearner: Reformulating Multimodal Machine Unlearning in the Era of Multimodal Large Language Modelscited by 29
  16. 2024To Forget or Not? Towards Practical Knowledge Unlearning for Large Language Modelscited by 25
  17. 2023Breaking the Trilemma of Privacy, Utility, Efficiency via Controllable Machine Unlearningcited by 21
  18. 2024RKLD: Reverse KL-Divergence-based Knowledge Distillation for Unlearning Personal Information in Large Language Modelscited by 21
  19. 2024A Probabilistic Perspective on Unlearning and Alignment for Large Language Modelscited by 21
  20. 2023MultiDelete for Multimodal Machine Unlearningcited by 20
  21. 2024CLIPErase: Efficient Unlearning of Visual-Textual Associations in CLIPcited by 19
  22. 2024Unlearning Traces the Influential Training Data of Language Modelscited by 18
  23. 2024Split, Unlearn, Merge: Leveraging Data Attributes for More Effective Unlearning in LLMscited by 18
  24. 2024Cross-Lingual Unlearning of Selective Knowledge in Multilingual Language Modelscited by 18
  25. 2024Second-Order Information Matters: Revisiting Machine Unlearning for Large Language Modelscited by 17
  26. 2024The Frontier of Data Erasure: Machine Unlearning for Large Language Modelscited by 17
  27. 2025Tool Unlearning for Tool-Augmented LLMscited by 17
  28. 2025GUARD: Generation-time LLM Unlearning via Adaptive Restriction and Detectioncited by 17
  29. 2024MU-Bench: A Multitask Multimodal Benchmark for Machine Unlearningcited by 16
  30. 2026ConeSep: Cone-based Robust Noise-Unlearning Compositional Network for Composed Image Retrievalcited by 15
  31. 2024UnStar: Unlearning with Self-Taught Anti-Sample Reasoning for LLMscited by 14
  32. 2024On the Limitations and Prospects of Machine Unlearning for Generative AIcited by 12
  33. 2024Negating Negatives: Alignment with Human Negative Samples via Distributional Dispreference Optimizationcited by 11
  34. 2025Unlearning vs. Obfuscation: Are We Truly Removing Knowledge?cited by 11
  35. 2025Distillation Robustifies Unlearningcited by 11
  36. 2024Machine Unlearning for Traditional Models and Large Language Models: A Short Surveycited by 10
  37. 2024Opt-Out: Investigating Entity-Level Unlearning for Large Language Models via Optimal Transportcited by 10
  38. 2025UIPE: Enhancing LLM Unlearning by Removing Knowledge Related to Forgetting Targetscited by 10

50 most cited of 94 All 94 (JSON)