mlsanityData to 7 Oct 2026

Machine Unlearning of Pre-trained Large Language Models

arXiv 2402.15159 ↗First posted 23 Feb 2024cs.CL cs.AI cs.CRcited by 118

On the map: LLM safety and security → Machine unlearning

Abstract (arXiv)

This study investigates the concept of the `right to be forgotten' within the context of large language models (LLMs). We explore machine unlearning as a pivotal solution, with a focus on pre-trained models--a notably under-researched area. Our research delineates a comprehensive framework for machine unlearning in pre-trained LLMs, encompassing a critical analysis of seven diverse unlearning methods. Through rigorous evaluation using curated datasets from arXiv, books, and GitHub, we establish a robust benchmark for unlearning performance, demonstrating that these methods are over 105 times more computationally efficient than retraining. Our results show that integrating gradient ascent with gradient descent on in-distribution data improves hyperparameter robustness. We also provide detailed guidelines for efficient hyperparameter tuning in the unlearning process. Our findings advance the discourse on ethical AI practices, offering substantive insights into the mechanics of machine unlearning for pre-trained LLMs and underscoring the potential for responsible AI development.

Key facts. Cited by 118 papers in the map (data through 7 Oct 2026). Placed in Machine unlearning, one of 12 directions in LLM safety and security. Builds on 50 of its 67 references that are in the map. First posted on arXiv 23 Feb 2024.

Builds on

50 of 67 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024Selective Forgetting: Advancing Machine Unlearning Techniques and Evaluation in Language Models8 Feb
  2. 2024TOFU: A Task of Fictitious Unlearning for LLMs11 Jan
  3. 2023Unlearn What You Want to Forget: Efficient Unlearning for LLMs31 Oct
  4. 2023Breaking the Trilemma of Privacy, Utility, Efficiency via Controllable Machine Unlearning28 Oct
  5. 2023Who's Harry Potter? Approximate Unlearning in LLMs3 Oct
  6. 2023Tight Bounds for Machine Unlearning via Differential Privacy2 Sep
  7. 2023Machine Unlearning Methodology base on Stochastic Teacher Network28 Aug
  8. 2023Learning to Unlearn: Instance-wise Unlearning for Pre-trained Classifiers27 Jan
  9. 2022Privacy Adhering Machine Un-learning in NLP19 Dec
  10. 2022Forget Unlearning: Towards True Data-Deletion in Machine Learning17 Oct
  11. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Models4 Oct
  12. 2022Can Bad Teaching Induce Forgetting? Unlearning in Deep Networks using an Incompetent Teacher17 May
3 more
  1. 2021Fast Yet Effective Machine Unlearning17 Nov
  2. 2021Adaptive Machine Unlearning8 Jun
  3. 2021Remember What You Want to Forget: Algorithms for Machine Unlearning4 Mar

Other references · 35

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  8. 2017Deep reinforcement learning from human preferencescited by 3,846
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2020Learning to summarize from human feedbackcited by 2,562
  11. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  12. 2020Extracting Training Data from Large Language Modelscited by 2,144
23 more
  1. 2019Machine Unlearningcited by 1,020
  2. 2023BloombergGPT: A Large Language Model for Financecited by 923
  3. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  4. 2021Membership Inference Attacks From First Principlescited by 839
  5. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  6. 2017Renyi Differential Privacycited by 643
  7. 2019Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networkscited by 599
  8. 2019Certified Data Removal from Machine Learning Modelscited by 512
  9. 2019Making AI Forget You: Data Deletion in Machine Learningcited by 447
  10. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  11. 2017Measuring Catastrophic Forgetting in Neural Networkscited by 359
  12. 2022Estimating the Carbon Footprint of BLOOM, a 176B Parameter Language Modelcited by 264
  13. 2020Approximate Data Deletion from Machine Learning Modelscited by 250
  14. 2020The Lipschitz Constant of Self-Attentioncited by 171
  15. 2022Measuring Forgetting of Memorized Training Examplescited by 118
  16. 2022Synthetic Text Generation with Differential Privacy: A Simple and Practical Recipecited by 108
  17. 2021Differential Privacy for Text Analytics via Natural Text Sanitizationcited by 74
  18. 2023DP-Forward: Fine-tuning and Inference on Language Models with Differential Privacy in Forward Passcited by 65
  19. 2020Formalizing Data Deletion in the Context of the Right to be Forgottencited by 46
  20. 2021SSSE: Efficiently Erasing Samples from Trained Machine Learning Modelscited by 28
  21. 2023PrivImage: Differentially Private Synthetic Image Generation using Diffusion Models with Semantic-Aware Pretrainingcited by 12
  22. 2023Towards Code Watermarking with Dual-Channel Transformationscited by 9
  23. 2023Secure Split Learning against Property Inference, Data Reconstruction, and Feature Space Hijacking Attackscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 27; 2025: 51; 2026: 40.

118 papers.

Sort:

By area:

  1. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearningcited by 484
  2. 2024Large Language Model Unlearning via Embedding-Corrupted Promptscited by 112
  3. 2024Min-K%++: Improved Baseline for Detecting Pre-Training Data from Large Language Modelscited by 102
  4. 2024RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Modelscited by 77
  5. 2024LLM Unlearning via Loss Adjustment with Only Forget Datacited by 71
  6. 2024SoK: Membership Inference Attacks on LLMs are Rushing Nowhere (and How to Fix It)cited by 54
  7. 2024A Closer Look at Machine Unlearning for Large Language Modelscited by 54
  8. 2024Machine Unlearning in Generative AI: A Surveycited by 49
  9. 2024Single Image Unlearning: Efficient Machine Unlearning in Multimodal Large Language Modelscited by 44
  10. 2025Unlearning Isn't Deletion: Investigating Reversibility of Machine Unlearning in LLMscited by 41
  11. 2024Meta-Unlearning on Diffusion Models: Preventing Relearning Unlearned Conceptscited by 39
  12. 2024Towards Robust and Parameter-Efficient Knowledge Unlearning for LLMscited by 36
38 more
  1. 2025Towards Label-Only Membership Inference Attack against Pre-trained Large Language Modelscited by 34
  2. 2024Offset Unlearning for Large Language Modelscited by 33
  3. 2024Machine Unlearning: A Comprehensive Surveycited by 33
  4. 2025A Comprehensive Survey of Machine Unlearning Techniques for Large Language Modelscited by 30
  5. 2024PrE-Text: Training Language Models on Private Federated Data in the Age of LLMscited by 29
  6. 2024Benchmarking Vision Language Model Unlearning via Fictitious Facial Identity Datasetcited by 29
  7. 2024WAGLE: Strategic Weight Attribution for Effective and Modular Unlearning in Large Language Modelscited by 25
  8. 2024When Machine Unlearning Meets Retrieval-Augmented Generation (RAG): Keep Secret or Forget Knowledge?cited by 20
  9. 2025Invariance Makes LLM Unlearning Resilient Even to Unanticipated Downstream Fine-Tuningcited by 20
  10. 2025Tool Unlearning for Tool-Augmented LLMscited by 17
  11. 2025GUARD: Generation-time LLM Unlearning via Adaptive Restriction and Detectioncited by 17
  12. 2024Towards Efficient and Effective Unlearning of Large Language Models for Recommendationcited by 16
  13. 2024Large Scale Knowledge Washingcited by 16
  14. 2026ConeSep: Cone-based Robust Noise-Unlearning Compositional Network for Composed Image Retrievalcited by 15
  15. 2025Do LLMs Really Forget? Evaluating Unlearning with Knowledge Correlation and Confidence Awarenesscited by 13
  16. 2024Evaluating Deep Unlearning in Large Language Modelscited by 12
  17. 2025SAUCE: Selective Concept Unlearning in Vision-Language Models with Sparse Autoencoderscited by 12
  18. 2024Mitigating Memorization In Language Modelscited by 11
  19. 2025Distillation Robustifies Unlearningcited by 11
  20. 2024Underestimated Privacy Risks for Minority Populations in Large Language Model Unlearningcited by 9
  21. 2025Proactive Privacy Amnesia for Large Language Models: Safeguarding PII with Negligible Impact on Model Utilitycited by 9
  22. 2025SoK: Machine Unlearning for Large Language Modelscited by 9
  23. 2025The Landscape of Memorization in LLMs: Mechanisms, Measurement, and Mitigationcited by 9
  24. 2025R-TOFU: Unlearning in Large Reasoning Modelscited by 8
  25. 2025LUNE: Efficient LLM Unlearning via LoRA Fine-Tuning with Negative Examplescited by 8
  26. 2024TrajDeleter: Enabling Trajectory Forgetting in Offline Reinforcement Learning Agentscited by 7
  27. 2025OFFSIDE: Benchmarking Unlearning Misinformation in Multimodal Large Language Modelscited by 7
  28. 2026MemPrivacy: Privacy-Preserving Personalized Memory Management for Edge-Cloud Agentscited by 7
  29. 2025PEBench: A Fictitious Dataset to Benchmark Machine Unlearning for Multimodal Large Language Modelscited by 6
  30. 2025Step-by-Step Reasoning Attack: Revealing 'Erased' Knowledge in Large Language Modelscited by 6
  31. 2026Agentic Unlearning: When LLM Agent Meets Machine Unlearningcited by 6
  32. 2026Attention Smoothing Is All You Need For Unlearningcited by 6
  33. 2025Position: Bridge the Gaps between Machine Unlearning and AI Regulationcited by 5
  34. 2025Erasing Without Remembering: Implicit Knowledge Forgetting in Large Language Modelscited by 5
  35. 2025PULSE: Practical Evaluation Scenarios for Large Multimodal Model Unlearningcited by 5
  36. 2025Erase or Hide? Suppressing Spurious Unlearning Neurons for Robust Unlearningcited by 5
  37. 2026Is Gradient Ascent Really Necessary? Memorize to Forget for Machine Unlearningcited by 5
  38. 2024Composable Interventions for Language Modelscited by 4

50 most cited of 118 All 118 (JSON)