mlsanityData to 7 Oct 2026

UNLEARN Efficient Removal of Knowledge in Large Language Models

arXiv 2408.04140 ↗First posted 8 Aug 2024cs.CL cs.AIcited by 9

Not on the map. Why?

Abstract (arXiv)

Given the prevalence of large language models (LLMs) and the prohibitive cost of training these models from scratch, dynamically forgetting specific knowledge e.g., private or proprietary, without retraining the model has become an important capability. This paper proposes a novel method to achieve this objective called UNLEARN. The approach builds upon subspace methods to identify and specifically target the removal of knowledge without adversely affecting other knowledge in the LLM. Results demonstrate 96% of targeted knowledge can be forgotten while maintaining performance on other knowledge within 2.5% of the original model, significantly outperforming the discriminatory abilities of the previous state-of-the-art. A dual method called LEARN is also proposed for targeted knowledge addition. Results show LEARN can match the fine-tuning accuracy of Low-Rank Adaptation (LoRA) without adversely affecting similar tasks.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 29 of its 42 references that are in the map. First posted on arXiv 8 Aug 2024.

Builds on

29 of 42 references.

References · 29

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  9. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  10. 2019Machine Unlearningcited by 1,020
  11. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  12. 2017The NarrativeQA Reading Comprehension Challengecited by 590
17 more
  1. 2023Erasing Concepts from Diffusion Modelscited by 530
  2. 2024TOFU: A Task of Fictitious Unlearning for LLMscited by 507
  3. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  4. 2022Memory-Based Model Editing at Scalecited by 443
  5. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Modelscited by 424
  6. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 376
  7. 2023SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generationcited by 376
  8. 2023Who's Harry Potter? Approximate Unlearning in LLMscited by 366
  9. 2023Unlearn What You Want to Forget: Efficient Unlearning for LLMscited by 211
  10. 2021Adaptive Machine Unlearningcited by 164
  11. 2023Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attackscited by 156
  12. 2024Towards Safer Large Language Models through Machine Unlearningcited by 134
  13. 2022The Right to be Forgotten in Federated Learning: An Efficient Realization with Rapid Retrainingcited by 109
  14. 2023KGA: A General Machine Unlearning Framework Based on Knowledge Gap Alignmentcited by 94
  15. 2024Guardrail Baselines for Unlearning in LLMscited by 90
  16. 2022Input-Tuning: Adapting Unfamiliar Inputs to Frozen Pretrained Modelscited by 34
  17. 2016Domain Adaptation of Recurrent Neural Networks for Natural Language Understandingcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 4; 2026: 2.

9 papers.

  1. 2024Low-Rank Adaptation for Foundation Models: A Comprehensive Reviewcited by 35
  2. 2024Gradient Routing: Masking Gradients to Localize Computation in Neural Networkscited by 28
  3. 2025Emotions Where Art Thou: Understanding and Characterizing the Emotional Latent Space of Large Language Modelscited by 14
  4. 2025Position: Bridge the Gaps between Machine Unlearning and AI Regulationcited by 5
  5. 2025Forgetting: A New Mechanism Towards Better Large Language Model Fine-tuningcited by 4
  6. 2025PBFT-Backed Semantic Voting for Multi-Agent Memory Pruningcited by 1
  7. 2024ResidualDroppath: Enhancing Feature Reuse over Residual Connectionscited by 0
  8. 2026Machine Unlearning in Low-Dimensional Feature Subspacecited by 0
  9. 2026LLM Unlearning for Cyber Defense: A Survey on Methods, Challenges, and Emerging Threatscited by 0