mlsanityData to 14 Oct 2026

GRU: Mitigating the Trade-off between Unlearning and Retention for LLMs

arXiv 2503.09117 ↗First posted 12 Mar 2025cs.LG cs.CLcited by 12

On the map: LLM safety and security → Machine unlearning

Abstract (arXiv)

Large language model (LLM) unlearning has demonstrated its essential role in removing privacy and copyright-related responses, crucial for their legal and safe applications. However, the pursuit of complete unlearning often comes with substantial costs due to its compromises in their general functionality, leading to a notorious trade-off between unlearning and retention. It motivates this paper to explore enhanced unlearning schemes that can mitigate this trade-off. Specifically, we propose Gradient Rectified Unlearning (GRU), an improved framework that regulates the directions of gradient updates during the unlearning procedure such that their side impacts on other, unrelated responses can be minimized. GRU is easy and general to implement, demonstrating practical effectiveness across a variety of well-established unlearning benchmarks.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in Machine unlearning, one of 12 directions in LLM safety and security. Builds on 35 of its 41 references that are in the map. First posted on arXiv 12 Mar 2025.

Builds on

35 of 41 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025Rethinking LLM Unlearning Objectives: A Gradient Perspective and Go Beyond26 Feb
  2. 2024Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning9 Oct
  3. 2024MUSE: Machine Unlearning Six-Way Evaluation for Language Models8 Jul
  4. 2024To Each (Textual Sequence) Its Own: Improving Memorized-Data Unlearning in Large Language Models6 May
  5. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning8 Apr
  6. 2024Rethinking Machine Unlearning for Large Language Models13 Feb
  7. 2024TOFU: A Task of Fictitious Unlearning for LLMs11 Jan
  8. 2023Large Language Model Unlearning14 Oct
  9. 2023Who's Harry Potter? Approximate Unlearning in LLMs3 Oct
  10. 2023Right to be Forgotten in the Era of Large Language Models: Implications, Challenges, and Solutions8 Jul

Other references · 25

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2023Qwen Technical Reportcited by 3,344
  8. 2023A Survey of Large Language Modelscited by 2,576
  9. 2020Extracting Training Data from Large Language Modelscited by 2,189
  10. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  11. 2019Language Models as Knowledge Bases?cited by 1,936
  12. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
13 more
  1. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  2. 2020Gradient Surgery for Multi-Task Learningcited by 1,256
  3. 2022Editing Models with Task Arithmeticcited by 1,120
  4. 2023BloombergGPT: A Large Language Model for Financecited by 939
  5. 2021Robust fine-tuning of zero-shot modelscited by 797
  6. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 623
  7. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 509
  8. 2023Zephyr: Direct Distillation of LM Alignmentcited by 506
  9. 2023Llemma: An Open Language Model For Mathematicscited by 388
  10. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 387
  11. 2024Security and Privacy Challenges of Large Language Models: A Surveycited by 186
  12. 2023In-context Pretraining: Language Modeling Beyond Document Boundariescited by 79
  13. 2023Robust Generalization against Photon-Limited Corruptions via Worst-Case Sharpness Minimizationcited by 29

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 7.

12 papers.

  1. 2025OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metricscited by 83
  2. 2025Exploring Criteria of Loss Reweighting to Enhance LLM Unlearningcited by 39
  3. 2025Model Unlearning via Sparse Autoencoder Subspace Guided Projectionscited by 18
  4. 2025LLM Unlearning with LLM Beliefscited by 17
  5. 2026AEGIS: Adversarial Target-Guided Retention-Data-Free Robust Concept Erasure from Diffusion Modelscited by 5
  6. 2026REBEL: Hidden Knowledge Recovery via Evolutionary-Based Evaluation Loopcited by 4
  7. 2026Visual-Guided Key-Token Regularization for Multimodal Large Language Model Unlearningcited by 3
  8. 2025Ready2Unlearn: A Learning-Time Approach for Preparing Models with Future Unlearning Readinesscited by 2
  9. 2026Per-parameter Task Arithmetic for Unlearning in Large Language Modelscited by 2
  10. 2026KUDA: Knowledge Unlearning by Deviating Representation for Large Language Modelscited by 2
  11. 2026EvoMU: Evolutionary Machine Unlearningcited by 0
  12. 2026MeGU: Machine-Guided Unlearning with Target Feature Disentanglementcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025GRAIL: Gradient-Based Adaptive Unlearning for Privacy and Copyright in LLMs
  2. 2026Unlearning in LLMs: Methods, Evaluation, and Open Challenges
  3. 2024Position: LLM Unlearning Benchmarks are Weak Measures of Progress
  4. 2025DP2Unlearning: An Efficient and Guaranteed Unlearning Framework for LLMs
  5. 2025A General Framework to Enhance Fine-tuning-based LLM Unlearning
  6. 2024Avoiding Copyright Infringement via Large Language Model Unlearning
  7. 2024Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Difference
  8. 2024A Closer Look at Machine Unlearning for Large Language Models