mlsanityData to 14 Oct 2026

OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metrics

arXiv 2506.12618 ↗First posted 14 Jun 2025cs.CLcited by 83

On the map: LLM safety and security → Machine unlearning

Abstract (arXiv)

Robust unlearning is crucial for safely deploying large language models (LLMs) in environments where data privacy, model safety, and regulatory compliance must be ensured. Yet the task is inherently challenging, partly due to difficulties in reliably measuring whether unlearning has truly occurred. Moreover, fragmentation in current methodologies and inconsistent evaluation metrics hinder comparative analysis and reproducibility. To unify and accelerate research efforts, we introduce OpenUnlearning, a standardized and extensible framework designed explicitly for benchmarking both LLM unlearning methods and metrics. OpenUnlearning integrates 13 unlearning algorithms and 16 diverse evaluations across 3 leading benchmarks (TOFU, MUSE, and WMDP) and also enables analyses of forgetting behaviors across 450+ checkpoints we publicly release. Leveraging OpenUnlearning, we propose a novel meta-evaluation benchmark focused specifically on assessing the faithfulness and robustness of evaluation metrics themselves. We also benchmark diverse unlearning methods and provide a comparative analysis against an extensive evaluation suite. Overall, we establish a clear, community-driven pathway toward rigorous development in LLM unlearning research.

Key facts. Cited by 83 papers in the map (data through 14 Oct 2026). Placed in Machine unlearning, one of 12 directions in LLM safety and security. Builds on 60 of its 82 references that are in the map. First posted on arXiv 14 Jun 2025.

Builds on

60 of 82 references.

Closest ancestors · 38

Same direction, newest first

  1. 2025Constrained Entropic Unlearning: A Primal-Dual Framework for Large Language Models5 Jun
  2. 2025Exploring Criteria of Loss Reweighting to Enhance LLM Unlearning17 May
  3. 2025SemEval-2025 Task 4: Unlearning sensitive content from Large Language Models2 Apr
  4. 2025GRU: Mitigating the Trade-off between Unlearning and Retention for LLMs12 Mar
  5. 2025Are We Truly Forgetting? A Critical Re-examination of Machine Unlearning Evaluation Protocols10 Mar
  6. 2025CE-U: Cross Entropy Unlearning3 Mar
  7. 2025Rethinking LLM Unlearning Objectives: A Gradient Perspective and Go Beyond26 Feb
  8. 2025LUME: LLM Unlearning with Multitask Evaluations20 Feb
  9. 2025ReLearn: Unlearning via Learning for Large Language Models16 Feb
  10. 2025Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyond7 Feb
  11. 2024Unified Parameter-Efficient Unlearning for LLMs30 Nov
  12. 2024Does Unlearning Truly Unlearn? A Black Box Evaluation of LLM Unlearning Methods18 Nov
26 more
  1. 2024Extracting Unlearned Information from LLMs with Activation Steering4 Nov
  2. 2024Catastrophic Failure of LLM Unlearning via Quantization21 Oct
  3. 2024LLM Unlearning via Loss Adjustment with Only Forget Data14 Oct
  4. 2024Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning9 Oct
  5. 2024A Probabilistic Perspective on Unlearning and Alignment for Large Language Models4 Oct
  6. 2024Position: LLM Unlearning Benchmarks are Weak Measures of Progress3 Oct
  7. 2024Erasing Conceptual Knowledge from Language Models3 Oct
  8. 2024An Adversarial Perspective on Machine Unlearning for AI Safety26 Sep
  9. 2024Alternate Preference Optimization for Unlearning Factual Knowledge in Large Language Models20 Sep
  10. 2024MEOW: MEMOry Supervised LLM Unlearning Via Inverted Facts18 Sep
  11. 2024MUSE: Machine Unlearning Six-Way Evaluation for Language Models8 Jul
  12. 2024To Forget or Not? Towards Practical Knowledge Unlearning for Large Language Models2 Jul
  13. 2024Unlearning or Obfuscating? Jogging the Memory of Unlearned LLMs via Benign Relearning19 Jun
  14. 2024Opt-Out: Investigating Entity-Level Unlearning for Large Language Models via Optimal Transport18 Jun
  15. 2024Soft Prompting for Unlearning in Large Language Models17 Jun
  16. 2024RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Models16 Jun
  17. 2024Towards Effective Evaluations and Comparisons for LLM Unlearning Methods13 Jun
  18. 2024SOUL: Unlocking the Power of Second-Order Optimization for LLM Unlearning28 Apr
  19. 2024Rethinking LLM Memorization through the Lens of Adversarial Compression23 Apr
  20. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning8 Apr
  21. 2024Eight Methods to Evaluate Robust Unlearning in LLMs26 Feb
  22. 2024UNDIAL: Self-Distillation with Adjusted Logits for Robust Unlearning in Large Language Models15 Feb
  23. 2024Rethinking Machine Unlearning for Large Language Models13 Feb
  24. 2024TOFU: A Task of Fictitious Unlearning for LLMs11 Jan
  25. 2023Who's Harry Potter? Approximate Unlearning in LLMs3 Oct
  26. 2022A Survey of Machine Unlearning6 Sep

Other references · 22

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  6. 2020Extracting Training Data from Large Language Modelscited by 2,189
  7. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  8. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  9. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 916
  10. 2022Quantifying Memorization Across Neural Language Modelscited by 763
  11. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 623
  12. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 509
10 more
  1. 2023Zephyr: Direct Distillation of LM Alignmentcited by 506
  2. 2023Detecting Pretraining Data from Large Language Modelscited by 387
  3. 2023DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Modelscited by 245
  4. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 220
  5. 2024Do Membership Inference Attacks Work on Large Language Models?cited by 195
  6. 2024Nemotron-4 340B Technical Reportcited by 144
  7. 2023Copyright Violations and Large Language Modelscited by 108
  8. 2024Ethos: Rectifying Language Models in Orthogonal Parameter Spacecited by 20
  9. 2024Pandora's White-Box: Precise Training Data Detection and Extraction in Large Language Modelscited by 11
  10. 2024The Boy Who Survived: Removing Harry Potter from an LLM is harder than reportedcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 14; 2026: 69.

83 papers.

Sort:

By area:

  1. 2025LLM Unlearning with LLM Beliefscited by 17
  2. 2025Unlearning That Lasts: Utility-Preserving, Robust, and Almost Irreversible Forgetting in LLMscited by 11
  3. 2026Maximizing Local Entropy Where It Matters: Prefix-Aware Localized LLM Unlearningcited by 10
  4. 2025Leak@k: Unlearning Does Not Make LLMs Forget Under Probabilistic Decodingcited by 9
  5. 2026FML-bench: A Controlled Study of AI Research Agent Strategies from the Perspective of Search Dynamicscited by 8
  6. 2025Model Collapse Is Not a Bug but a Feature in Machine Unlearning for LLMscited by 7
  7. 2026Contrastive Representation Shaping for LLM Unlearningcited by 7
  8. 2026Attention Smoothing Is All You Need For Unlearningcited by 7
  9. 2026Explainable LLM Unlearning Through Reasoningcited by 7
  10. 2025Unlearning Comparator: A Visual Analytics System for Comparative Evaluation of Machine Unlearning Methodscited by 4
  11. 2025Geometric-disentangelment Unlearningcited by 4
  12. 2026Reinforcement Unlearning via Group Relative Policy Optimizationcited by 4
38 more
  1. 2025Forget to Know, Remember to Use: Context-Aware Unlearning for Large Language Modelscited by 3
  2. 2026Visual-Guided Key-Token Regularization for Multimodal Large Language Model Unlearningcited by 3
  3. 2025Oblivionis: A Lightweight Learning and Unlearning Framework for Federated Large Language Modelscited by 2
  4. 2025Direct Token Optimization: A Self-contained Approach to Large Language Model Unlearningcited by 2
  5. 2026From Domains to Instances: Dual-Granularity Data Synthesis for LLM Unlearningcited by 2
  6. 2026A Mechanistic Perspective and Circuit-Guided Difficulty Metric for Unlearningcited by 2
  7. 2026Per-parameter Task Arithmetic for Unlearning in Large Language Modelscited by 2
  8. 2026CALIBURN: Self-Calibrated LLM Unlearning Alignmentcited by 2
  9. 2026Forget What Matters, Keep the Rest: Selective Unlearning of Informative Tokenscited by 2
  10. 2026Inference-Time Machine Unlearning via Gated Activation Redirectioncited by 2
  11. 2026Distinguishable Deletion: Unifying Knowledge Erasure and Refusal for Large Language Model Unlearningcited by 2
  12. 2026Divergence Decoding: Inference-Time Unlearning via Auxiliary Modelscited by 2
  13. 2026Multilingual Unlearning in LLMs: Transfer, Dynamics, and Reversibilitycited by 2
  14. 2026PreUnlearn: Auditing Collateral Knowledge Damage Before Large Language Model Unlearningcited by 2
  15. 2026LACUNA: A Testbed for Evaluating Localization Precision for LLM Unlearningcited by 2
  16. 2025IMU: Influence-guided Machine Unlearningcited by 1
  17. 2025Unlearning at Scale: State-Exact Trace-Preserving Deletion in Billion-Parameter Language Modelscited by 1
  18. 2025On the importance of multiple training seeds for evaluating machine unlearningcited by 1
  19. 2025RippleBench: Capturing Ripple Effects Using Existing Knowledge Repositoriescited by 1
  20. 2026Temper-Then-Tilt: Principled Unlearning for Generative Models through Tempering and Classifier Guidancecited by 1
  21. 2026Metric Unreliability in Multimodal Machine Unlearning: A Systematic Analysis and Principled Unified Scorecited by 1
  22. 2026Measuring the Depth of LLM Unlearning via Activation Patchingcited by 1
  23. 2026Learning What to Forget: Improving LLM Unlearning via Learned Token-Level Importancecited by 1
  24. 2026Unlearning as Distribution Restoration: A Controlled Counterfactual Study, a Validated Selective Screen, and the Limits of Oracle-Free Certificationcited by 1
  25. 2026Beyond Binary Rewards: A Comparative Study of Reward Design for Reinforcement Unlearningcited by 1
  26. 2026GROM: Gradient-Free Rapid One-Shot Machine Unlearningcited by 1
  27. 2026Measure, Don't Optimize: Forecasting Recovery in LLM Unlearningcited by 1
  28. 2026Can LLMs Truly Forget? Revealing Unlearning Gaps Through Adversarial Evaluationcited by 1
  29. 2026Beyond Cross-Lingual Transfer: Benchmarking Propagation Boundaries in Multilingual LLM Unlearningcited by 1
  30. 2025REMIND: Input Loss Landscapes Reveal Residual Memorization in Post-Unlearning LLMscited by 0
  31. 2026SAU: Sparsity-Aware Unlearning for LLMs via Gradient Masking and Importance Redistributioncited by 0
  32. 2026EvoMU: Evolutionary Machine Unlearningcited by 0
  33. 2026MPU: Towards Secure and Privacy-Preserving Knowledge Unlearning for Large Language Modelscited by 0
  34. 2026WIN-U: Woodbury-Informed Newton-Unlearning as a retain-free Machine Unlearning Frameworkcited by 0
  35. 2026Unlearning What Matters: Token-Level Attribution for Precise Language Model Unlearningcited by 0
  36. 2026Retain-Neutral Surrogates for Min-Max Unlearningcited by 0
  37. 2026SHRED: Retain-Set-Free Unlearning via Self-Distillation with Logit Demotioncited by 0
  38. 2026MAAT: Multi-phase Adapter-Aware Targeted Unlearningcited by 0

50 most cited of 83 All 83 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Unlearning in LLMs: Methods, Evaluation, and Open Challenges
  2. 2025LLM Unlearning Reveals a Stronger-Than-Expected Coreset Effect in Current Benchmarks
  3. 2025LLM Unlearning Under the Microscope: A Full-Stack View on Methods and Metrics
  4. 2025A Comprehensive Survey of Machine Unlearning Techniques for Large Language Models
  5. 2026Open-MMUnlearning: Unifying Methods and Evaluation for MLLM Unlearning
  6. 2025BLUR: A Benchmark for LLM Unlearning Robust to Forget-Retain Overlap
  7. 2025A Survey on Unlearning in Large Language Models
  8. 2025LLM Unlearning Should Be Form-Independent