# OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metrics

arXiv: https://arxiv.org/abs/2506.12618 · HTML: https://mlsanity.com/paper/2506.12618/

## Key facts

- Cited by 83 papers in the map (data through 14 Oct 2026).
- Placed in Machine unlearning, one of 12 directions in LLM safety and security.
- Builds on 60 of its 82 references that are in the map.
- First posted on arXiv 14 Jun 2025.

## Placed in

- LLM safety and security / [Machine unlearning](https://mlsanity.com/safety/machine-unlearning/)

## Builds on (most cited first, 30 of 60)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [Gemma: Open Models Based on Gemini Research and Technology](https://mlsanity.com/paper/2403.08295/) (2024)
- [Quantifying Memorization Across Neural Language Models](https://mlsanity.com/paper/2202.07646/) (2022)
- [Eliciting Latent Predictions from Transformers with the Tuned Lens](https://mlsanity.com/paper/2303.08112/) (2023)
- [TOFU: A Task of Fictitious Unlearning for LLMs](https://mlsanity.com/paper/2401.06121/) (2024)
- [The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning](https://mlsanity.com/paper/2403.03218/) (2024)
- [Zephyr: Direct Distillation of LM Alignment](https://mlsanity.com/paper/2310.16944/) (2023)
- [Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning](https://mlsanity.com/paper/2404.05868/) (2024)
- [Detecting Pretraining Data from Large Language Models](https://mlsanity.com/paper/2310.16789/) (2023)
- [Who's Harry Potter? Approximate Unlearning in LLMs](https://mlsanity.com/paper/2310.02238/) (2023)
- [A Survey of Machine Unlearning](https://mlsanity.com/paper/2209.02299/) (2022)
- [MUSE: Machine Unlearning Six-Way Evaluation for Language Models](https://mlsanity.com/paper/2407.06460/) (2024)
- [Rethinking Machine Unlearning for Large Language Models](https://mlsanity.com/paper/2402.08787/) (2024)
- [DeepSpeed Ulysses: System Optimizations for Enabling Training of Extreme Long Sequence Transformer Models](https://mlsanity.com/paper/2309.14509/) (2023)
- [Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Models](https://mlsanity.com/paper/2205.10770/) (2022)
- [Do Membership Inference Attacks Work on Large Language Models?](https://mlsanity.com/paper/2402.07841/) (2024)
- [Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning](https://mlsanity.com/paper/2410.07163/) (2024)
- [Eight Methods to Evaluate Robust Unlearning in LLMs](https://mlsanity.com/paper/2402.16835/) (2024)
- [Nemotron-4 340B Technical Report](https://mlsanity.com/paper/2406.11704/) (2024)
- [An Adversarial Perspective on Machine Unlearning for AI Safety](https://mlsanity.com/paper/2409.18025/) (2024)
- [Copyright Violations and Large Language Models](https://mlsanity.com/paper/2310.13771/) (2023)
- [SOUL: Unlocking the Power of Second-Order Optimization for LLM Unlearning](https://mlsanity.com/paper/2404.18239/) (2024)
- [Rethinking LLM Memorization through the Lens of Adversarial Compression](https://mlsanity.com/paper/2404.15146/) (2024)

## Built on by (most cited first, 30 of 83)

- [LLM Unlearning with LLM Beliefs](https://mlsanity.com/paper/2510.19422/) (2025)
- [Unlearning That Lasts: Utility-Preserving, Robust, and Almost Irreversible Forgetting in LLMs](https://mlsanity.com/paper/2509.02820/) (2025)
- [Maximizing Local Entropy Where It Matters: Prefix-Aware Localized LLM Unlearning](https://mlsanity.com/paper/2601.03190/) (2026)
- [Leak@k: Unlearning Does Not Make LLMs Forget Under Probabilistic Decoding](https://mlsanity.com/paper/2511.04934/) (2025)
- [FML-bench: A Controlled Study of AI Research Agent Strategies from the Perspective of Search Dynamics](https://mlsanity.com/paper/2605.17373/) (2026)
- [Model Collapse Is Not a Bug but a Feature in Machine Unlearning for LLMs](https://mlsanity.com/paper/2507.04219/) (2025)
- [Contrastive Representation Shaping for LLM Unlearning](https://mlsanity.com/paper/2601.22028/) (2026)
- [Attention Smoothing Is All You Need For Unlearning](https://mlsanity.com/paper/2603.01285/) (2026)
- [Explainable LLM Unlearning Through Reasoning](https://mlsanity.com/paper/2603.09980/) (2026)
- [Unlearning Comparator: A Visual Analytics System for Comparative Evaluation of Machine Unlearning Methods](https://mlsanity.com/paper/2508.12730/) (2025)
- [Geometric-disentangelment Unlearning](https://mlsanity.com/paper/2511.17100/) (2025)
- [Reinforcement Unlearning via Group Relative Policy Optimization](https://mlsanity.com/paper/2601.20568/) (2026)
- [Forget to Know, Remember to Use: Context-Aware Unlearning for Large Language Models](https://mlsanity.com/paper/2510.17620/) (2025)
- [Visual-Guided Key-Token Regularization for Multimodal Large Language Model Unlearning](https://mlsanity.com/paper/2601.22020/) (2026)
- [Oblivionis: A Lightweight Learning and Unlearning Framework for Federated Large Language Models](https://mlsanity.com/paper/2508.08875/) (2025)
- [Direct Token Optimization: A Self-contained Approach to Large Language Model Unlearning](https://mlsanity.com/paper/2510.00125/) (2025)
- [From Domains to Instances: Dual-Granularity Data Synthesis for LLM Unlearning](https://mlsanity.com/paper/2601.04278/) (2026)
- [A Mechanistic Perspective and Circuit-Guided Difficulty Metric for Unlearning](https://mlsanity.com/paper/2601.09624/) (2026)
- [Per-parameter Task Arithmetic for Unlearning in Large Language Models](https://mlsanity.com/paper/2601.22030/) (2026)
- [CALIBURN: Self-Calibrated LLM Unlearning Alignment](https://mlsanity.com/paper/2602.02824/) (2026)
- [Forget What Matters, Keep the Rest: Selective Unlearning of Informative Tokens](https://mlsanity.com/paper/2604.17785/) (2026)
- [Inference-Time Machine Unlearning via Gated Activation Redirection](https://mlsanity.com/paper/2605.12765/) (2026)
- [Distinguishable Deletion: Unifying Knowledge Erasure and Refusal for Large Language Model Unlearning](https://mlsanity.com/paper/2605.16776/) (2026)
- [Divergence Decoding: Inference-Time Unlearning via Auxiliary Models](https://mlsanity.com/paper/2605.31293/) (2026)
- [Multilingual Unlearning in LLMs: Transfer, Dynamics, and Reversibility](https://mlsanity.com/paper/2606.03291/) (2026)
- [PreUnlearn: Auditing Collateral Knowledge Damage Before Large Language Model Unlearning](https://mlsanity.com/paper/2606.18473/) (2026)
- [LACUNA: A Testbed for Evaluating Localization Precision for LLM Unlearning](https://mlsanity.com/paper/2607.02513/) (2026)
- [IMU: Influence-guided Machine Unlearning](https://mlsanity.com/paper/2508.01620/) (2025)
- [Unlearning at Scale: State-Exact Trace-Preserving Deletion in Billion-Parameter Language Models](https://mlsanity.com/paper/2508.12220/) (2025)
- [On the importance of multiple training seeds for evaluating machine unlearning](https://mlsanity.com/paper/2510.26714/) (2025)

## Abstract (arXiv)

Robust unlearning is crucial for safely deploying large language models (LLMs) in environments where data privacy, model safety, and regulatory compliance must be ensured. Yet the task is inherently challenging, partly due to difficulties in reliably measuring whether unlearning has truly occurred. Moreover, fragmentation in current methodologies and inconsistent evaluation metrics hinder comparative analysis and reproducibility. To unify and accelerate research efforts, we introduce OpenUnlearning, a standardized and extensible framework designed explicitly for benchmarking both LLM unlearning methods and metrics. OpenUnlearning integrates 13 unlearning algorithms and 16 diverse evaluations across 3 leading benchmarks (TOFU, MUSE, and WMDP) and also enables analyses of forgetting behaviors across 450+ checkpoints we publicly release. Leveraging OpenUnlearning, we propose a novel meta-evaluation benchmark focused specifically on assessing the faithfulness and robustness of evaluation metrics themselves. We also benchmark diverse unlearning methods and provide a comparative analysis against an extensive evaluation suite. Overall, we establish a clear, community-driven pathway toward rigorous development in LLM unlearning research.