mlsanityData to 7 Oct 2026

An Adversarial Perspective on Machine Unlearning for AI Safety

arXiv 2409.18025 ↗First posted 26 Sep 2024cs.LG cs.AI cs.CLcited by 113

On the map: LLM safety and security → Machine unlearning

Abstract (arXiv)

Large language models are finetuned to refuse questions about hazardous knowledge, but these protections can often be bypassed. Unlearning methods aim at completely removing hazardous capabilities from models and make them inaccessible to adversaries. This work challenges the fundamental differences between unlearning and traditional safety post-training from an adversarial perspective. We demonstrate that existing jailbreak methods, previously reported as ineffective against unlearning, can be successful when applied carefully. Furthermore, we develop a variety of adaptive methods that recover most supposedly unlearned capabilities. For instance, we show that finetuning on 10 unrelated examples or removing specific directions in the activation space can recover most hazardous capabilities for models edited with RMU, a state-of-the-art unlearning method. Our findings challenge the robustness of current unlearning approaches and question their advantages over safety training.

Key facts. Cited by 113 papers in the map (data through 7 Oct 2026). Placed in Machine unlearning, one of 12 directions in LLM safety and security. Builds on 61 of its 76 references that are in the map. First posted on arXiv 26 Sep 2024.

Builds on

61 of 76 references.

Closest ancestors · 18

Same direction, newest first

  1. 2024On Effects of Steering Latent Representation for Large Language Model Unlearning12 Aug
  2. 2024MUSE: Machine Unlearning Six-Way Evaluation for Language Models8 Jul
  3. 2024UnUnlearning: Unlearning is not sufficient for content regulation in advanced generative AI27 Jun
  4. 2024Machine Unlearning Fails to Remove Data Poisoning Attacks25 Jun
  5. 2024Split, Unlearn, Merge: Leveraging Data Attributes for More Effective Unlearning in LLMs17 Jun
  6. 2024Intrinsic Test of Unlearning Using Parametric Knowledge Traces17 Jun
  7. 2024RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Models16 Jun
  8. 2024Large Language Model Unlearning via Embedding-Corrupted Prompts12 Jun
  9. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning8 Apr
  10. 2024Eight Methods to Evaluate Robust Unlearning in LLMs26 Feb
  11. 2024Towards Safer Large Language Models through Machine Unlearning15 Feb
  12. 2024Rethinking Machine Unlearning for Large Language Models13 Feb
6 more
  1. 2024TOFU: A Task of Fictitious Unlearning for LLMs11 Jan
  2. 2023Unlearn What You Want to Forget: Efficient Unlearning for LLMs31 Oct
  3. 2023Large Language Model Unlearning14 Oct
  4. 2023Who's Harry Potter? Approximate Unlearning in LLMs3 Oct
  5. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Models4 Oct
  6. 2022Towards Adversarial Evaluations for Inexact Machine Unlearning17 Jan

Other references · 43

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2016Pointer Sentinel Mixture Modelscited by 2,992
  7. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  8. 2024Mixtral of Expertscited by 1,785
  9. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  10. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  11. 2019Machine Unlearningcited by 1,020
  12. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
31 more
  1. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  2. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  3. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  4. 2020On Adaptive Attacks to Adversarial Example Defensescited by 547
  5. 2017Adversarial Examples Are Not Easily Detected: Bypassing Ten Detection Methodscited by 542
  6. 2020BERT-ATTACK: Adversarial Attack Against BERT Using BERTcited by 488
  7. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  8. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  9. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  10. 2020BAE: BERT-based Adversarial Examples for Text Classificationcited by 316
  11. 2024Improving Alignment and Robustness with Circuit Breakerscited by 302
  12. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 284
  13. 2019Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Themcited by 268
  14. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 220
  15. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulationcited by 202
  16. 2024Foundational Challenges in Assuring Alignment and Safety of Large Language Modelscited by 193
  17. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70Bcited by 183
  18. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 179
  19. 2023Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attackscited by 156
  20. 2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 132
  21. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 110
  22. 2023Copyright Violations and Large Language Modelscited by 108
  23. 2024Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Spacecited by 88
  24. 2024Fast Adversarial Attacks on Language Models In One GPU Minutecited by 85
  25. 2023DEPN: Detecting and Editing Privacy Neurons in Pretrained Language Modelscited by 75
  26. 2023Unveiling the Implicit Toxicity in Large Language Modelscited by 51
  27. 2021Data Poisoning Won't Save You From Facial Recognitioncited by 45
  28. 2024Evaluating Copyright Takedown Methods for Language Modelscited by 42
  29. 2024Adversarial Perturbations Cannot Reliably Protect Artists From Generative AIcited by 38
  30. 2024Unlearning or Concealment? A Critical Analysis and Evaluation Metrics for Unlearning in Diffusion Modelscited by 12
  31. 2024FLRT: Fluent Student-Teacher Redteamingcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 62; 2026: 39.

113 papers.

Sort:

By area:

  1. 2025The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses Against Llm Jailbreaks and Prompt Injectionscited by 108
  2. 2025OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metricscited by 80
  3. 2025Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyondcited by 76
  4. 2024Do Unlearning Methods Remove Information from Language Model Weights?cited by 66
  5. 2024Applying sparse autoencoders to unlearn knowledge in language modelscited by 58
  6. 2024On Evaluating the Durability of Safeguards for Open-Weight LLMscited by 51
  7. 2024Meta-Unlearning on Diffusion Models: Preventing Relearning Unlearned Conceptscited by 39
  8. 2025Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilitiescited by 36
  9. 2025SafeEraser: Enhancing Safety in Multimodal Large Language Models through Multimodal Machine Unlearningcited by 31
  10. 2025A Comprehensive Survey of Machine Unlearning Techniques for Large Language Modelscited by 30
  11. 2024Gradient Routing: Masking Gradients to Localize Computation in Neural Networkscited by 28
  12. 2025Representation Bending for Large Language Model Safetycited by 28
38 more
  1. 2024Does Unlearning Truly Unlearn? A Black Box Evaluation of LLM Unlearning Methodscited by 25
  2. 2025Modality-Aware Neuron Pruning for Unlearning in Multimodal Large Language Modelscited by 25
  3. 2025BLUR: A Bi-Level Optimization Approach for LLM Unlearningcited by 23
  4. 2025Cyber-Zero: Training Cybersecurity Agents without Runtimecited by 22
  5. 2025SAEs Can Improve Unlearning: Dynamic Sparse Autoencoder Guardrails for Precision Unlearning in LLMscited by 21
  6. 2025Invariance Makes LLM Unlearning Resilient Even to Unanticipated Downstream Fine-Tuningcited by 20
  7. 2024Exploring Memorization and Copyright Violation in Frontier LLMs: A Study of the New York Times v. OpenAI 2023 Lawsuitcited by 17
  8. 2025Tool Unlearning for Tool-Augmented LLMscited by 17
  9. 2025Training Language Model Agents to Find Vulnerabilities with CTF-Dojocited by 15
  10. 2024Unstable Unlearning: The Hidden Risk of Concept Resurgence in Diffusion Modelscited by 14
  11. 2024Machine Unlearning Doesn't Do What You Think: Lessons for Generative AI Policy and Researchcited by 14
  12. 2025Position: Adversarial ML for LLMs Is Not Making Any Progresscited by 14
  13. 2025Exact Unlearning of Finetuning Data via Model Merging at Scalecited by 14
  14. 2025RULE: Reinforcement UnLEarning Achieves Forget-Retain Pareto Optimalitycited by 14
  15. 2024Evaluating Deep Unlearning in Large Language Modelscited by 12
  16. 2025From Dormant to Deleted: Tamper-Resistant Unlearning Through Weight-Space Regularizationcited by 12
  17. 2025BLUR: A Benchmark for LLM Unlearning Robust to Forget-Retain Overlapcited by 12
  18. 2025Episodic memory in AI agents poses risks that should be studied and mitigatedcited by 11
  19. 2025LLM Unlearning Reveals a Stronger-Than-Expected Coreset Effect in Current Benchmarkscited by 11
  20. 2025CTRAP: Embedding Collapse Trap to Safeguard Large Language Models from Harmful Fine-Tuningcited by 11
  21. 2025Precise In-Parameter Concept Erasure in Large Language Modelscited by 11
  22. 2025Distillation Robustifies Unlearningcited by 11
  23. 2026Shaping capabilities with token-level data filteringcited by 11
  24. 2025Self-Destructive Language Modelcited by 10
  25. 2025Unlearning Isn't Invisible: Detecting Unlearning Traces in LLMs from Model Outputscited by 10
  26. 2025UCD: Unlearning in LLMs via Contrastive Decodingcited by 9
  27. 2025STREAM (ChemBio): A Standard for Transparently Reporting Evaluations in AI Model Reportscited by 9
  28. 2025Unlearning That Lasts: Utility-Preserving, Robust, and Almost Irreversible Forgetting in LLMscited by 9
  29. 2025Breaking Bad Tokens: Detoxification of LLMs Using Sparse Autoencoderscited by 8
  30. 2025On Jailbreaking Quantized Language Models Through Fault Injection Attackscited by 8
  31. 2025Beyond Data Filtering: Knowledge Localization for Capability Removal in LLMscited by 8
  32. 2025Dynamic Risk Assessments for Offensive Cybersecurity Agentscited by 7
  33. 2025Does Machine Unlearning Truly Remove Knowledge?cited by 7
  34. 2025Existing Large Language Model Unlearning Evaluations Are Inconclusivecited by 7
  35. 2025Watch the Weights: Unsupervised monitoring and control of fine-tuned LLMscited by 7
  36. 2024PII-Scope: A Comprehensive Study on Training Data PII Extraction Attacks in LLMscited by 6
  37. 2024Copyright-Protected Language Generation via Adaptive Model Fusioncited by 6
  38. 2025Improving LLM Unlearning Robustness via Random Perturbationscited by 6

50 most cited of 113 All 113 (JSON)