# LUME: LLM Unlearning with Multitask Evaluations

arXiv: https://arxiv.org/abs/2502.15097 · HTML: https://mlsanity.com/paper/2502.15097/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in Machine unlearning, one of 12 directions in LLM safety and security.
- Builds on 12 of its 32 references that are in the map.
- First posted on arXiv 20 Feb 2025.

## Placed in

- LLM safety and security / [Machine unlearning](https://mlsanity.com/safety/machine-unlearning/)

## Builds on (most cited first, 12 of 12)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [OLMo: Accelerating the Science of Language Models](https://mlsanity.com/paper/2402.00838/) (2024)
- [TOFU: A Task of Fictitious Unlearning for LLMs](https://mlsanity.com/paper/2401.06121/) (2024)
- [Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning](https://mlsanity.com/paper/2404.05868/) (2024)
- [Who's Harry Potter? Approximate Unlearning in LLMs](https://mlsanity.com/paper/2310.02238/) (2023)
- [MUSE: Machine Unlearning Six-Way Evaluation for Language Models](https://mlsanity.com/paper/2407.06460/) (2024)
- [In-Context Unlearning: Language Models as Few Shot Unlearners](https://mlsanity.com/paper/2310.07579/) (2023)
- [Unlearn What You Want to Forget: Efficient Unlearning for LLMs](https://mlsanity.com/paper/2310.20150/) (2023)
- [What makes unlearning hard and what to do about it](https://mlsanity.com/paper/2406.01257/) (2024)
- [Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Difference](https://mlsanity.com/paper/2406.08607/) (2024)
- [Unlearning as multi-task optimization: A normalized gradient difference approach with an adaptive learning rate](https://mlsanity.com/paper/2410.22086/) (2024)

## Built on by (most cited first, 22 of 22)

- [A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deployment](https://mlsanity.com/paper/2504.15585/) (2025)
- [OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metrics](https://mlsanity.com/paper/2506.12618/) (2025)
- [GUARD: Generation-time LLM Unlearning via Adaptive Restriction and Detection](https://mlsanity.com/paper/2505.13312/) (2025)
- [SoK: Machine Unlearning for Large Language Models](https://mlsanity.com/paper/2506.09227/) (2025)
- [Does Localization Inform Unlearning? A Rigorous Examination of Local Parameter Attribution for Knowledge Unlearning in Language Models](https://mlsanity.com/paper/2505.16252/) (2025)
- [Leak@k: Unlearning Does Not Make LLMs Forget Under Probabilistic Decoding](https://mlsanity.com/paper/2511.04934/) (2025)
- [Not Every Token Needs Forgetting: Selective Unlearning to Limit Change in Utility in Large Language Model Unlearning](https://mlsanity.com/paper/2506.00876/) (2025)
- [AILS-NTUA at SemEval-2025 Task 4: Parameter-Efficient Unlearning for Large Language Models using Data Chunking](https://mlsanity.com/paper/2503.02443/) (2025)
- [Machine Unlearning under Overparameterization](https://mlsanity.com/paper/2505.22601/) (2025)
- [A Survey on Unlearning in Large Language Models](https://mlsanity.com/paper/2510.25117/) (2025)
- [SemEval-2025 Task 4: Unlearning sensitive content from Large Language Models](https://mlsanity.com/paper/2504.02883/) (2025)
- [Atyaephyra at SemEval-2025 Task 4: Low-Rank Negative Preference Optimization](https://mlsanity.com/paper/2503.13690/) (2025)
- [ZJUKLAB at SemEval-2025 Task 4: Unlearning via Model Merging](https://mlsanity.com/paper/2503.21088/) (2025)
- [Lacuna Inc. at SemEval-2025 Task 4: LoRA-Enhanced Influence-Based Unlearning for LLMs](https://mlsanity.com/paper/2506.04044/) (2025)
- [Cyber for AI at SemEval-2025 Task 4: Forgotten but Not Lost: The Balancing Act of Selective Unlearning in Large Language Models](https://mlsanity.com/paper/2503.04795/) (2025)
- [SHA256 at SemEval-2025 Task 4: Selective Amnesia -- Constrained Unlearning for Large Language Models via Knowledge Isolation](https://mlsanity.com/paper/2504.12996/) (2025)
- [Mr. Snuffleupagus at SemEval-2025 Task 4: Unlearning Factual Knowledge from LLMs Using Adaptive RMU](https://mlsanity.com/paper/2506.16548/) (2025)
- [iShumei-Chinchunmei at SemEval-2025 Task 4: A balanced forgetting and retention multi-task framework using effective unlearning loss](https://mlsanity.com/paper/2507.16263/) (2025)
- [RapidUn: Influence-Driven Parameter Reweighting for Efficient Large Language Model Unlearning](https://mlsanity.com/paper/2512.04457/) (2025)
- [Lethe: Principled Dual-Stream Update for Persistent Knowledge Erasure in Federated Unlearning](https://mlsanity.com/paper/2601.22601/) (2026)
- [Forget Narrowly, Retain Broadly: Unlearning as an Asymmetric Generalization Problem](https://mlsanity.com/paper/2607.09236/) (2026)
- [Linguistic Loopholes in LLM Unlearning: From a 174-Language Benchmark to Coverage-Aware Unlearning](https://mlsanity.com/paper/2609.40286/) (2026)

## Abstract (arXiv)

Unlearning aims to remove copyrighted, sensitive, or private content from large language models (LLMs) without a full retraining. In this work, we develop a multi-task unlearning benchmark (LUME) which features three tasks: (1) unlearn synthetically generated creative short novels, (2) unlearn synthetic biographies with sensitive information, and (3) unlearn a collection of public biographies. We further release two fine-tuned LLMs of 1B and 7B parameter sizes as the target models. We conduct detailed evaluations of several recently proposed unlearning algorithms and present results on carefully crafted metrics to understand their behavior and limitations.