mlsanityData to 14 Oct 2026

Stress Testing Unlearning Algorithms

arXiv 2608.22527 ↗First posted 23 Aug 2026cs.LGcited by 0

On the map: LLM safety and security → Machine unlearning

Abstract (arXiv)

Recently, machine unlearning, the removal of specific training data influence from a model, has gained increasing attention. In large language models (LLMs), unlearning is particularly challenging due to the ambiguity of inputs and outputs. Con- sequently, rigorous evaluation is critical for assessing both safety and utility, and for driving progress in unlearning meth- ods. We identify two key shortcomings in existing unlearning benchmarks: (1) they do not actively test whether unlearned information can still be forcibly extracted, and (2) they fail to evaluate performance preservation on boundary questions, be- nign queries that are semantically close to the unlearned con- tent. Here we introduce WMDP++, an extension of WMDP that addresses these gaps by incorporating targeted extrac- tion of unlearned information and systematic evaluation on boundary questions. WMDP++ provides a more stringent and informative benchmark for evaluating unlearning in LLMs.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Machine unlearning, one of 12 directions in LLM safety and security. Builds on 18 of its 27 references that are in the map. First posted on arXiv 23 Aug 2026.

Builds on

18 of 27 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025CRISP: Persistent Concept Unlearning via Sparse Autoencoders19 Aug
  2. 2025BLUR: A Benchmark for LLM Unlearning Robust to Forget-Retain Overlap28 May
  3. 2025Go Beyond Your Means: Unlearning with Per-Sample Gradient Orthogonalization4 Mar
  4. 2024Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning9 Oct
  5. 2024Position: LLM Unlearning Benchmarks are Weak Measures of Progress3 Oct
  6. 2024An Adversarial Perspective on Machine Unlearning for AI Safety26 Sep
  7. 2024MUSE: Machine Unlearning Six-Way Evaluation for Language Models8 Jul
  8. 2024UnUnlearning: Unlearning is not sufficient for content regulation in advanced generative AI27 Jun
  9. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning8 Apr
  10. 2024TOFU: A Task of Fictitious Unlearning for LLMs11 Jan

Other references · 8

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  3. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  4. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  5. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 509
  6. 2023Zephyr: Direct Distillation of LM Alignmentcited by 506
  7. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 390
  8. 2025Dialogue Injection Attack: Jailbreaking LLMs through Context Manipulationcited by 14

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025LLM Unlearning Under the Microscope: A Full-Stack View on Methods and Metrics
  2. 2026The Unlearning Mirage: A Dynamic Framework for Evaluating LLM Unlearning
  3. 2025OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metrics
  4. 2024Towards Effective Evaluations and Comparisons for LLM Unlearning Methods
  5. 2025LLM Unlearning Reveals a Stronger-Than-Expected Coreset Effect in Current Benchmarks
  6. 2025The Erasure Illusion: Stress-Testing the Generalization of LLM Forgetting Evaluation
  7. 2024Does Unlearning Truly Unlearn? A Black Box Evaluation of LLM Unlearning Methods
  8. 2025A Comprehensive Survey of Machine Unlearning Techniques for Large Language Models