# Not Every Token Needs Forgetting: Selective Unlearning to Limit Change in Utility in Large Language Model Unlearning

arXiv: https://arxiv.org/abs/2506.00876 · HTML: https://mlsanity.com/paper/2506.00876/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Machine unlearning, one of 12 directions in LLM safety and security.
- Builds on 13 of its 25 references that are in the map.
- First posted on arXiv 1 Jun 2025.

## Placed in

- LLM safety and security / [Machine unlearning](https://mlsanity.com/safety/machine-unlearning/)

## Builds on (most cited first, 13 of 13)

- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [TOFU: A Task of Fictitious Unlearning for LLMs](https://mlsanity.com/paper/2401.06121/) (2024)
- [Knowledge Unlearning for Mitigating Privacy Risks in Language Models](https://mlsanity.com/paper/2210.01504/) (2022)
- [MUSE: Machine Unlearning Six-Way Evaluation for Language Models](https://mlsanity.com/paper/2407.06460/) (2024)
- [Unlearn What You Want to Forget: Efficient Unlearning for LLMs](https://mlsanity.com/paper/2310.20150/) (2023)
- [Rho-1: Not All Tokens Are What You Need](https://mlsanity.com/paper/2404.07965/) (2024)
- [Selective Forgetting: Advancing Machine Unlearning Techniques and Evaluation in Language Models](https://mlsanity.com/paper/2402.05813/) (2024)
- [Unlearning as multi-task optimization: A normalized gradient difference approach with an adaptive learning rate](https://mlsanity.com/paper/2410.22086/) (2024)
- [LUME: LLM Unlearning with Multitask Evaluations](https://mlsanity.com/paper/2502.15097/) (2025)
- [Unveiling Entity-Level Unlearning for Large Language Models: A Comprehensive Analysis](https://mlsanity.com/paper/2406.15796/) (2024)

## Built on by (most cited first, 7 of 7)

- [Maximizing Local Entropy Where It Matters: Prefix-Aware Localized LLM Unlearning](https://mlsanity.com/paper/2601.03190/) (2026)
- [Not All Tokens Are Meant to Be Forgotten](https://mlsanity.com/paper/2506.03142/) (2025)
- [Bergson: An Open Source Library for Data Attribution](https://mlsanity.com/paper/2606.11660/) (2026)
- [STaR: Sensitive Trajectory Regulation for Unlearning in Large Reasoning Models](https://mlsanity.com/paper/2601.09281/) (2026)
- [A Mechanistic Perspective and Circuit-Guided Difficulty Metric for Unlearning](https://mlsanity.com/paper/2601.09624/) (2026)
- [Unlearning What Matters: Token-Level Attribution for Precise Language Model Unlearning](https://mlsanity.com/paper/2605.00364/) (2026)
- [What to Forget in Unlearning? Forget Set Curation for Language Models](https://mlsanity.com/paper/2608.14855/) (2026)

## Abstract (arXiv)

Large Language Model (LLM) unlearning has recently gained significant attention, driven by the need to remove unwanted information, such as private, sensitive, or copyrighted content, from LLMs. However, conventional unlearning approaches indiscriminately update model parameters to forget all tokens in a target document, including common tokens (e.g., pronouns, prepositions, general nouns) that carry general knowledge. In this paper, we highlight that not every token needs forgetting. We propose Selective Unlearning (SU), which identifies a critical subset of tokens within the forgetting set that is relevant to the unwanted information, and unlearns only those tokens. Experiments on two benchmarks and six baseline unlearning algorithms demonstrate that SU not only achieves effective unlearning on the targeted forget data, but also significantly preserves the model's utility in the retaining set.