mlsanityData to 14 Oct 2026

Towards Evaluation for Real-World LLM Unlearning

arXiv 2508.01324 ↗First posted 2 Aug 2025cs.AIcited by 2

On the map: LLM safety and security → Machine unlearning

Abstract (arXiv)

This paper analyzes the limitations of existing unlearning evaluation metrics in terms of practicality, exactness, and robustness in real-world LLM unlearning scenarios. To overcome these limitations, we propose a new metric called Distribution Correction-based Unlearning Evaluation (DCUE). It identifies core tokens and corrects distributional biases in their confidence scores using a validation set. The evaluation results are quantified using the Kolmogorov-Smirnov test. Experimental results demonstrate that DCUE overcomes the limitations of existing metrics, which also guides the design of more practical and reliable unlearning algorithms in the future.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Machine unlearning, one of 12 directions in LLM safety and security. Builds on 31 of its 40 references that are in the map. First posted on arXiv 2 Aug 2025.

Builds on

31 of 40 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning9 Oct
  2. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning8 Apr
  3. 2024Guardrail Baselines for Unlearning in LLMs5 Mar
  4. 2024Eight Methods to Evaluate Robust Unlearning in LLMs26 Feb
  5. 2024TOFU: A Task of Fictitious Unlearning for LLMs11 Jan
  6. 2023ERASER: Machine Unlearning in MLaaS via an Inference Serving-Aware Approach3 Nov
  7. 2023Unlearn What You Want to Forget: Efficient Unlearning for LLMs31 Oct
  8. 2023In-Context Unlearning: Language Models as Few Shot Unlearners11 Oct
  9. 2023Ticketed Learning-Unlearning Schemes27 Jun
  10. 2022Privacy Adhering Machine Un-learning in NLP19 Dec
  11. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Models4 Oct
  12. 2022Quark: Controllable Text Generation with Reinforced Unlearning26 May
2 more
  1. 2021Adaptive Machine Unlearning8 Jun
  2. 2021Remember What You Want to Forget: Algorithms for Machine Unlearning4 Mar

Other references · 17

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  2. 2020Extracting Training Data from Large Language Modelscited by 2,189
  3. 2019Machine Unlearningcited by 1,061
  4. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 509
  5. 2019Making AI Forget You: Data Deletion in Machine Learningcited by 465
  6. 2020Approximate Data Deletion from Machine Learning Modelscited by 255
  7. 2023Large Language Models in Finance: A Surveycited by 226
  8. 2023Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attackscited by 158
  9. 2023Composing Parameter-Efficient Modules with Arithmetic Operationscited by 156
  10. 2020DeltaGrad: Rapid retraining of machine learning modelscited by 149
  11. 2023LawBench: Benchmarking Legal Knowledge of Large Language Modelscited by 117
  12. 2023Foundation Models and Fair Usecited by 116
5 more
  1. 2023DEPN: Detecting and Editing Privacy Neurons in Pretrained Language Modelscited by 79
  2. 2023SILO Language Models: Isolating Legal Risk In a Nonparametric Datastorecited by 71
  3. 2023Knowledge Sanitization of Large Language Modelscited by 37
  4. 2024Fantastic Copyrighted Beasts and How (Not) to Generate Themcited by 24
  5. 2024Towards Efficient and Effective Unlearning of Large Language Models for Recommendationcited by 16

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025Towards Mitigating Excessive Forgetting in LLM Unlearning via Entanglement-Guidance with Proxy Constraintcited by 1
  2. 2026Adaptive and Explicit safe: Triggering Latent Safety Awareness in Large Reasoning Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Effective Evaluations and Comparisons for LLM Unlearning Methods
  2. 2024Does Unlearning Truly Unlearn? A Black Box Evaluation of LLM Unlearning Methods
  3. 2024Unlearning in- vs. out-of-distribution data in LLMs under gradient-based method
  4. 2025OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metrics
  5. 2025Do LLMs Really Forget? Evaluating Unlearning with Knowledge Correlation and Confidence Awareness
  6. 2025LLM Unlearning Reveals a Stronger-Than-Expected Coreset Effect in Current Benchmarks
  7. 2025LLM Unlearning Under the Microscope: A Full-Stack View on Methods and Metrics
  8. 2025LLM Unlearning Should Be Form-Independent