# Exploring Criteria of Loss Reweighting to Enhance LLM Unlearning

arXiv: https://arxiv.org/abs/2505.11953 · HTML: https://mlsanity.com/paper/2505.11953/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Placed in Machine unlearning, one of 12 directions in LLM safety and security.
- Builds on 67 of its 79 references that are in the map.
- First posted on arXiv 17 May 2025.

## Placed in

- LLM safety and security / [Machine unlearning](https://mlsanity.com/safety/machine-unlearning/)

## Builds on (most cited first, 30 of 67)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Editing Models with Task Arithmetic](https://mlsanity.com/paper/2212.04089/) (2022)
- [Machine Unlearning](https://mlsanity.com/paper/1912.03817/) (2019)
- [Aligning AI With Shared Human Values](https://mlsanity.com/paper/2008.02275/) (2020)
- [Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks](https://mlsanity.com/paper/1911.04933/) (2019)
- [Textbooks Are All You Need II: phi-1.5 technical report](https://mlsanity.com/paper/2309.05463/) (2023)
- [TOFU: A Task of Fictitious Unlearning for LLMs](https://mlsanity.com/paper/2401.06121/) (2024)
- [The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning](https://mlsanity.com/paper/2403.03218/) (2024)
- [Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning](https://mlsanity.com/paper/2404.05868/) (2024)
- [Safety Alignment Should Be Made More Than Just a Few Tokens Deep](https://mlsanity.com/paper/2406.05946/) (2024)
- [Knowledge Unlearning for Mitigating Privacy Risks in Language Models](https://mlsanity.com/paper/2210.01504/) (2022)
- [Detecting Pretraining Data from Large Language Models](https://mlsanity.com/paper/2310.16789/) (2023)
- [SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation](https://mlsanity.com/paper/2310.12508/) (2023)
- [Who's Harry Potter? Approximate Unlearning in LLMs](https://mlsanity.com/paper/2310.02238/) (2023)
- [DeepInception: Hypnotize Large Language Model to Be Jailbreaker](https://mlsanity.com/paper/2311.03191/) (2023)
- [Large Language Model Unlearning](https://mlsanity.com/paper/2310.10683/) (2023)
- [TrustLLM: Trustworthiness in Large Language Models](https://mlsanity.com/paper/2401.05561/) (2024)
- [Unrolling SGD: Understanding Factors Influencing Machine Unlearning](https://mlsanity.com/paper/2109.13398/) (2021)
- [White-box vs Black-box: Bayes Optimal Strategies for Membership Inference](https://mlsanity.com/paper/1908.11229/) (2019)
- [MUSE: Machine Unlearning Six-Way Evaluation for Language Models](https://mlsanity.com/paper/2407.06460/) (2024)
- [Rethinking Machine Unlearning for Large Language Models](https://mlsanity.com/paper/2402.08787/) (2024)
- [Learning with Noisy Labels Revisited: A Study Using Real-World Human Annotations](https://mlsanity.com/paper/2110.12088/) (2021)
- [Approximate Data Deletion from Machine Learning Models](https://mlsanity.com/paper/2002.10077/) (2020)
- [What is the Effect of Importance Weighting in Deep Learning?](https://mlsanity.com/paper/1812.03372/) (2018)

## Built on by (most cited first, 30 of 39)

- [OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metrics](https://mlsanity.com/paper/2506.12618/) (2025)
- [LLM Unlearning with LLM Beliefs](https://mlsanity.com/paper/2510.19422/) (2025)
- [Maximizing Local Entropy Where It Matters: Prefix-Aware Localized LLM Unlearning](https://mlsanity.com/paper/2601.03190/) (2026)
- [BalDRO: A Distributionally Robust Optimization based Framework for Large Language Model Unlearning](https://mlsanity.com/paper/2601.09172/) (2026)
- [VII: Visual Instruction Injection for Jailbreaking Image-to-Video Generation Models](https://mlsanity.com/paper/2602.20999/) (2026)
- [Is Gradient Ascent Really Necessary? Memorize to Forget for Machine Unlearning](https://mlsanity.com/paper/2602.06441/) (2026)
- [Downgrade to Upgrade: Optimizer Simplification Enhances Robustness in LLM Unlearning](https://mlsanity.com/paper/2510.00761/) (2025)
- [Geometric-disentangelment Unlearning](https://mlsanity.com/paper/2511.17100/) (2025)
- [REBEL: Hidden Knowledge Recovery via Evolutionary-Based Evaluation Loop](https://mlsanity.com/paper/2602.06248/) (2026)
- [Unifying Stable Optimization and Reference Regularization in RLHF](https://mlsanity.com/paper/2602.11523/) (2026)
- [Forget to Know, Remember to Use: Context-Aware Unlearning for Large Language Models](https://mlsanity.com/paper/2510.17620/) (2025)
- [Visual-Guided Key-Token Regularization for Multimodal Large Language Model Unlearning](https://mlsanity.com/paper/2601.22020/) (2026)
- [GUARD: Guided Unlearning and Retention via Data Attribution for Large Language Models](https://mlsanity.com/paper/2506.10946/) (2025)
- [Revisiting the Past: Data Unlearning with Model State History](https://mlsanity.com/paper/2506.20941/) (2025)
- [Direct Token Optimization: A Self-contained Approach to Large Language Model Unlearning](https://mlsanity.com/paper/2510.00125/) (2025)
- [A Mechanistic Perspective and Circuit-Guided Difficulty Metric for Unlearning](https://mlsanity.com/paper/2601.09624/) (2026)
- [Per-parameter Task Arithmetic for Unlearning in Large Language Models](https://mlsanity.com/paper/2601.22030/) (2026)
- [CALIBURN: Self-Calibrated LLM Unlearning Alignment](https://mlsanity.com/paper/2602.02824/) (2026)
- [Intellectual Property Protection for 3D Gaussian Splatting Assets: A Survey](https://mlsanity.com/paper/2602.03878/) (2026)
- [Forget What Matters, Keep the Rest: Selective Unlearning of Informative Tokens](https://mlsanity.com/paper/2604.17785/) (2026)
- [Inference-Time Machine Unlearning via Gated Activation Redirection](https://mlsanity.com/paper/2605.12765/) (2026)
- [Distinguishable Deletion: Unifying Knowledge Erasure and Refusal for Large Language Model Unlearning](https://mlsanity.com/paper/2605.16776/) (2026)
- [Towards Mitigating Excessive Forgetting in LLM Unlearning via Entanglement-Guidance with Proxy Constraint](https://mlsanity.com/paper/2508.20443/) (2025)
- [Temper-Then-Tilt: Principled Unlearning for Generative Models through Tempering and Classifier Guidance](https://mlsanity.com/paper/2602.10217/) (2026)
- [On the Robustness of Machine Unlearning for Vision-Language Models](https://mlsanity.com/paper/2605.26992/) (2026)
- [Learning What to Forget: Improving LLM Unlearning via Learned Token-Level Importance](https://mlsanity.com/paper/2606.06320/) (2026)
- [Rotation Control Unlearning: Quantifying and Controlling Continuous Unlearning for LLM with The Cognitive Rotation Space](https://mlsanity.com/paper/2509.25743/) (2025)
- [RapidUn: Influence-Driven Parameter Reweighting for Efficient Large Language Model Unlearning](https://mlsanity.com/paper/2512.04457/) (2025)
- [SAU: Sparsity-Aware Unlearning for LLMs via Gradient Masking and Importance Redistribution](https://mlsanity.com/paper/2602.00577/) (2026)
- [EvoMU: Evolutionary Machine Unlearning](https://mlsanity.com/paper/2602.02139/) (2026)

## Abstract (arXiv)

Loss reweighting has shown significant benefits for machine unlearning with large language models (LLMs). However, their exact functionalities are left unclear and the optimal strategy remains an open question, thus impeding the understanding and improvement of existing methodologies. In this paper, we identify two distinct goals of loss reweighting, namely, Saturation and Importance -- the former indicates that those insufficiently optimized data should be emphasized, while the latter stresses some critical data that are most influential for loss minimization. To study their usefulness, we design specific reweighting strategies for each goal and evaluate their respective effects on unlearning. We conduct extensive empirical analyses on well-established benchmarks, and summarize some important observations as follows: (i) Saturation enhances efficacy more than importance-based reweighting, and their combination can yield additional improvements. (ii) Saturation typically allocates lower weights to data with lower likelihoods, whereas importance-based reweighting does the opposite. (iii) The efficacy of unlearning is also largely influenced by the smoothness and granularity of the weight distributions. Based on these findings, we propose SatImp, a simple reweighting method that combines the advantages of both saturation and importance. Empirical results on extensive datasets validate the efficacy of our method, potentially bridging existing research gaps and indicating directions for future research. Our code is available at https://github.com/tmlr-group/SatImp.