mlsanityData to 7 Oct 2026

LLM Unlearning Under the Microscope: A Full-Stack View on Methods and Metrics

arXiv 2510.07626 ↗First posted 8 Oct 2025cs.LG cs.CLcited by 1

On the map: LLM safety and security → Machine unlearning

Abstract (arXiv)

Machine unlearning for large language models (LLMs) aims to remove undesired data, knowledge, and behaviors (e.g., for safety, privacy, or copyright) while preserving useful model capabilities. Despite rapid progress over the past two years, research in LLM unlearning remains fragmented, with limited clarity on what constitutes effective unlearning and how it should be rigorously evaluated. In this work, we present a principled taxonomy of twelve recent stateful unlearning methods, grouped into three methodological families: divergence-driven optimization, representation misalignment, and rejection-based targeted unlearning. Building on this taxonomy, we revisit the evaluation of unlearning effectiveness (UE), utility retention (UT), and robustness (Rob), focusing on the WMDP benchmark. Our analysis shows that current evaluations, dominated by multiple-choice question (MCQ) accuracy, offer only a narrow perspective, often overstating success while overlooking the model's actual generation behavior. To address this gap, we introduce open question-answering (Open-QA) metrics that better capture generative performance and reveal the inherent UE-UT tradeoff across method families. Furthermore, we demonstrate that robustness requires finer-grained analysis: for example, vulnerabilities differ substantially between in-domain relearning and out-of-domain fine-tuning, even though both fall under model-level attacks. Through this study, we hope to deliver a full-stack revisit of LLM unlearning and actionable guidance for designing and evaluating future methods.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Machine unlearning, one of 12 directions in LLM safety and security. Builds on 64 of its 69 references that are in the map. First posted on arXiv 8 Oct 2025.

Builds on

64 of 69 references.

Closest ancestors · 28

Same direction, newest first

  1. 2025Unlearning That Lasts: Utility-Preserving, Robust, and Almost Irreversible Forgetting in LLMs2 Sep
  2. 2025Reasoning Model Unlearning: Forgetting Traces, Not Just Answers, While Preserving Reasoning Skills15 Jun
  3. 2025Robust LLM Unlearning with MUDMAN: Meta-Unlearning with Disruption Masking And Normalization14 Jun
  4. 2025Distillation Robustifies Unlearning6 Jun
  5. 2025Invariance Makes LLM Unlearning Resilient Even to Unanticipated Downstream Fine-Tuning2 Jun
  6. 2025Existing Large Language Model Unlearning Evaluations Are Inconclusive31 May
  7. 2025BLUR: A Benchmark for LLM Unlearning Robust to Forget-Retain Overlap28 May
  8. 2025Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyond7 Feb
  9. 2025Open Problems in Machine Unlearning for AI Safety9 Jan
  10. 2024Catastrophic Failure of LLM Unlearning via Quantization21 Oct
  11. 2024Do Unlearning Methods Remove Information from Language Model Weights?11 Oct
  12. 2024A Closer Look at Machine Unlearning for Large Language Models10 Oct
16 more
  1. 2024Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning9 Oct
  2. 2024Erasing Conceptual Knowledge from Language Models3 Oct
  3. 2024An Adversarial Perspective on Machine Unlearning for AI Safety26 Sep
  4. 2024Learning to Refuse: Towards Mitigating Privacy Risks in LLMs14 Jul
  5. 2024MUSE: Machine Unlearning Six-Way Evaluation for Language Models8 Jul
  6. 2024RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Models16 Jun
  7. 2024Large Language Model Unlearning via Embedding-Corrupted Prompts12 Jun
  8. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning8 Apr
  9. 2024Eight Methods to Evaluate Robust Unlearning in LLMs26 Feb
  10. 2024Towards Safer Large Language Models through Machine Unlearning15 Feb
  11. 2024Rethinking Machine Unlearning for Large Language Models13 Feb
  12. 2024TOFU: A Task of Fictitious Unlearning for LLMs11 Jan
  13. 2023SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation19 Oct
  14. 2023Large Language Model Unlearning14 Oct
  15. 2022Quark: Controllable Text Generation with Reinforced Unlearning26 May
  16. 2021On the Necessity of Auditable Algorithmic Definitions for Machine Unlearning22 Oct

Other references · 36

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  7. 2016Pointer Sentinel Mixture Modelscited by 2,992
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  9. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  10. 2019Invariant Risk Minimizationcited by 1,832
  11. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  12. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
24 more
  1. 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
  2. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  3. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  4. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  5. 2019MathQA: Towards Interpretable Math Word Problem Solving with Operation-Based Formalismscited by 653
  6. 2019Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networkscited by 599
  7. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 488
  8. 2024Improving Alignment and Robustness with Circuit Breakerscited by 302
  9. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 252
  10. 2018Adversarial Logit Pairingcited by 224
  11. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 220
  12. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70Bcited by 183
  13. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
  14. 2023Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attackscited by 156
  15. 2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 132
  16. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 110
  17. 2024Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Spacecited by 88
  18. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 86
  19. 2024Fast Adversarial Attacks on Language Models In One GPU Minutecited by 85
  20. 2023DEPN: Detecting and Editing Privacy Neurons in Pretrained Language Modelscited by 75
  21. 2024Covert Malicious Finetuning: Challenges in Safeguarding LLM Adaptationcited by 74
  22. 2025Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilitiescited by 36
  23. 2018Evaluating and Understanding the Robustness of Adversarial Logit Pairingcited by 34
  24. 2024Accurate and Nuanced Open-QA Evaluation Through Textual Entailmentcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Subspace Control: Turning Constrained Model Steering into Controllable Spectral Optimizationcited by 1