mlsanityData to 7 Oct 2026

LUME: LLM Unlearning with Multitask Evaluations

arXiv 2502.15097 ↗First posted 20 Feb 2025cs.CL cs.LGcited by 21

On the map: LLM safety and security → Machine unlearning

Abstract (arXiv)

Unlearning aims to remove copyrighted, sensitive, or private content from large language models (LLMs) without a full retraining. In this work, we develop a multi-task unlearning benchmark (LUME) which features three tasks: (1) unlearn synthetically generated creative short novels, (2) unlearn synthetic biographies with sensitive information, and (3) unlearn a collection of public biographies. We further release two fine-tuned LLMs of 1B and 7B parameter sizes as the target models. We conduct detailed evaluations of several recently proposed unlearning algorithms and present results on carefully crafted metrics to understand their behavior and limitations.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Placed in Machine unlearning, one of 12 directions in LLM safety and security. Builds on 12 of its 32 references that are in the map. First posted on arXiv 20 Feb 2025.

Builds on

12 of 32 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024Unlearning as multi-task optimization: A normalized gradient difference approach with an adaptive learning rate29 Oct
  2. 2024MUSE: Machine Unlearning Six-Way Evaluation for Language Models8 Jul
  3. 2024Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Difference12 Jun
  4. 2024What makes unlearning hard and what to do about it3 Jun
  5. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning8 Apr
  6. 2024TOFU: A Task of Fictitious Unlearning for LLMs11 Jan
  7. 2023Unlearn What You Want to Forget: Efficient Unlearning for LLMs31 Oct
  8. 2023In-Context Unlearning: Language Models as Few Shot Unlearners11 Oct
  9. 2023Who's Harry Potter? Approximate Unlearning in LLMs3 Oct

Other references · 3

  1. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  2. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  3. 2024OLMo: Accelerating the Science of Language Modelscited by 599

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 19; 2026: 2.

21 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  2. 2025OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metricscited by 80
  3. 2025GUARD: Generation-time LLM Unlearning via Adaptive Restriction and Detectioncited by 17
  4. 2025Does Localization Inform Unlearning? A Rigorous Examination of Local Parameter Attribution for Knowledge Unlearning in Language Modelscited by 10
  5. 2025SoK: Machine Unlearning for Large Language Modelscited by 9
  6. 2025Leak@k: Unlearning Does Not Make LLMs Forget Under Probabilistic Decodingcited by 9
  7. 2025Not Every Token Needs Forgetting: Selective Unlearning to Limit Change in Utility in Large Language Model Unlearningcited by 7
  8. 2025AILS-NTUA at SemEval-2025 Task 4: Parameter-Efficient Unlearning for Large Language Models using Data Chunkingcited by 6
  9. 2025Machine Unlearning under Overparameterizationcited by 4
  10. 2025SemEval-2025 Task 4: Unlearning sensitive content from Large Language Modelscited by 3
  11. 2025Atyaephyra at SemEval-2025 Task 4: Low-Rank Negative Preference Optimizationcited by 2
  12. 2025ZJUKLAB at SemEval-2025 Task 4: Unlearning via Model Mergingcited by 2
9 more
  1. 2025A Survey on Unlearning in Large Language Modelscited by 2
  2. 2025Lacuna Inc. at SemEval-2025 Task 4: LoRA-Enhanced Influence-Based Unlearning for LLMscited by 1
  3. 2025Cyber for AI at SemEval-2025 Task 4: Forgotten but Not Lost: The Balancing Act of Selective Unlearning in Large Language Modelscited by 0
  4. 2025SHA256 at SemEval-2025 Task 4: Selective Amnesia -- Constrained Unlearning for Large Language Models via Knowledge Isolationcited by 0
  5. 2025Mr. Snuffleupagus at SemEval-2025 Task 4: Unlearning Factual Knowledge from LLMs Using Adaptive RMUcited by 0
  6. 2025iShumei-Chinchunmei at SemEval-2025 Task 4: A balanced forgetting and retention multi-task framework using effective unlearning losscited by 0
  7. 2025RapidUn: Influence-Driven Parameter Reweighting for Efficient Large Language Model Unlearningcited by 0
  8. 2026Lethe: Principled Dual-Stream Update for Persistent Knowledge Erasure in Federated Unlearningcited by 0
  9. 2026Linguistic Loopholes in LLM Unlearning: From a 174-Language Benchmark to Coverage-Aware Unlearningcited by 0