mlsanityData to 7 Oct 2026

WaterDrum: Watermarking for Data-centric Unlearning Metric

arXiv 2505.05064 ↗First posted 8 May 2025cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Large language model (LLM) unlearning is critical in real-world applications where it is necessary to efficiently remove the influence of private, copyrighted, or harmful data from some users. Existing utility-centric unlearning metrics (based on model utility) may fail to accurately evaluate the extent of unlearning in realistic settings such as when the forget and retain sets have semantically similar content and/or retraining the model from scratch on the retain set is impractical. This paper presents the first data-centric unlearning metric for LLMs called WaterDrum that exploits robust text watermarking to overcome these limitations. We introduce new benchmark datasets (with different levels of data similarity) for LLM unlearning that can be used to rigorously evaluate unlearning algorithms via WaterDrum. Our code is available at https://github.com/lululu008/WaterDrum and our new benchmark datasets are released at https://huggingface.co/datasets/Glow-AI/WaterDrum-Ax.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 35 of its 53 references that are in the map. First posted on arXiv 8 May 2025.

Builds on

35 of 53 references.

References · 35

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  3. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  4. 2022Editing Models with Task Arithmeticcited by 1,076
  5. 2019Machine Unlearningcited by 1,020
  6. 2023A Watermark for Large Language Modelscited by 667
  7. 2019Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networkscited by 599
  8. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  9. 2024TOFU: A Task of Fictitious Unlearning for LLMscited by 507
  10. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 488
  11. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Modelscited by 424
  12. 2023Detecting Pretraining Data from Large Language Modelscited by 375
23 more
  1. 2022A Survey of Machine Unlearningcited by 298
  2. 2023Robust Distortion-free Watermarks for Language Modelscited by 279
  3. 2023Towards Unbounded Machine Unlearningcited by 261
  4. 2024MUSE: Machine Unlearning Six-Way Evaluation for Language Modelscited by 251
  5. 2024Rethinking Machine Unlearning for Large Language Modelscited by 247
  6. 2021Fast Yet Effective Machine Unlearningcited by 209
  7. 2022Can Bad Teaching Induce Forgetting? Unlearning in Deep Networks using an Incompetent Teachercited by 189
  8. 2024Do Membership Inference Attacks Work on Large Language Models?cited by 188
  9. 2020Mixed-Privacy Forgetting in Deep Networkscited by 158
  10. 2024Eight Methods to Evaluate Robust Unlearning in LLMscited by 153
  11. 2022Zero-Shot Machine Unlearningcited by 125
  12. 2024Machine Unlearning of Pre-trained Large Language Modelscited by 118
  13. 2023A Survey of Text Watermarking in the Era of Large Language Modelscited by 102
  14. 2023Robust Multi-bit Natural Language Watermarking through Invariant Featurescited by 76
  15. 2024Position: LLM Unlearning Benchmarks are Weak Measures of Progresscited by 58
  16. 2021Tracing Text Provenance via Context-Aware Lexical Substitutioncited by 40
  17. 2022Evaluating Machine Unlearning via Epistemic Uncertaintycited by 39
  18. 2023Knowledge Unlearning for LLMs: Tasks, Methods, and Challengescited by 37
  19. 2024Towards Effective Evaluations and Comparisons for LLM Unlearning Methodscited by 27
  20. 2024PostMark: A Robust Blackbox Watermark for Large Language Modelscited by 24
  21. 2023Embarrassingly Simple Text Watermarkscited by 23
  22. 2023Watermarking LLMs with Weight Quantizationcited by 19
  23. 2024Evaluating Deep Unlearning in Large Language Modelscited by 12

Built on by

Not cited yet.