# Learning to Unlearn while Retaining: Combating Gradient Conflicts in Machine Unlearning

arXiv: https://arxiv.org/abs/2503.06339 · HTML: https://mlsanity.com/paper/2503.06339/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in Machine unlearning, one of 12 directions in LLM safety and security.
- Builds on 60 of its 83 references that are in the map.
- First posted on arXiv 8 Mar 2025.

## Placed in

- LLM safety and security / [Machine unlearning](https://mlsanity.com/safety/machine-unlearning/)

## Builds on (most cited first, 30 of 60)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [SNIP: Single-shot Network Pruning based on Connection Sensitivity](https://mlsanity.com/paper/1810.02340/) (2018)
- [MaskGAN: Towards Diverse and Interactive Facial Image Manipulation](https://mlsanity.com/paper/1907.11922/) (2019)
- [Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks](https://mlsanity.com/paper/1911.04933/) (2019)
- [Erasing Concepts from Diffusion Models](https://mlsanity.com/paper/2303.07345/) (2023)
- [TOFU: A Task of Fictitious Unlearning for LLMs](https://mlsanity.com/paper/2401.06121/) (2024)
- [Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Models](https://mlsanity.com/paper/2211.05105/) (2022)
- [Unified Concept Editing in Diffusion Models](https://mlsanity.com/paper/2308.14761/) (2023)
- [SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation](https://mlsanity.com/paper/2310.12508/) (2023)
- [Large Language Model Unlearning](https://mlsanity.com/paper/2310.10683/) (2023)
- [Forget-Me-Not: Learning to Forget in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.17591/) (2023)
- [Unrolling SGD: Understanding Factors Influencing Machine Unlearning](https://mlsanity.com/paper/2109.13398/) (2021)
- [Towards Unbounded Machine Unlearning](https://mlsanity.com/paper/2302.09880/) (2023)
- [Gradient Matching for Domain Generalization](https://mlsanity.com/paper/2104.09937/) (2021)
- [Approximate Data Deletion from Machine Learning Models](https://mlsanity.com/paper/2002.10077/) (2020)
- [Machine Unlearning of Features and Labels](https://mlsanity.com/paper/2108.11577/) (2021)
- [Model Sparsity Can Simplify Machine Unlearning](https://mlsanity.com/paper/2304.04934/) (2023)
- [To Generate or Not? Safety-Driven Unlearned Diffusion Models Are Still Easy To Generate Unsafe Images ... For Now](https://mlsanity.com/paper/2310.11868/) (2023)
- [Selective Amnesia: A Continual Learning Approach to Forgetting in Deep Generative Models](https://mlsanity.com/paper/2305.10120/) (2023)
- [Defensive Unlearning with Adversarial Training for Robust Concept Erasure in Diffusion Models](https://mlsanity.com/paper/2405.15234/) (2024)
- [The Power of Interpolation: Understanding the Effectiveness of SGD in Modern Over-parametrized Learning](https://mlsanity.com/paper/1712.06559/) (2017)
- [Graph Unlearning](https://mlsanity.com/paper/2103.14991/) (2021)

## Built on by (most cited first, 19 of 19)

- [Machine Unlearning under Retain-Forget Entanglement](https://mlsanity.com/paper/2603.26569/) (2026)
- [RAZOR: Ratio-Aware Layer Editing for Targeted Unlearning in Vision Transformers and Diffusion Models](https://mlsanity.com/paper/2603.14819/) (2026)
- [CURE:Circuit-Aware Unlearning for LLM-based Recommendation](https://mlsanity.com/paper/2604.04982/) (2026)
- [Locality-Aware Continual Unlearning for Diffusion Models](https://mlsanity.com/paper/2512.02657/) (2025)
- [Models Designed to Forget: Machine Unlearning via Key Deletion](https://mlsanity.com/paper/2603.15033/) (2026)
- [Coeff-Tuning: A Graph Filter Subspace View for Tuning Attention-Based Large Models](https://mlsanity.com/paper/2503.18337/) (2025)
- [GIFT: Gradient-aware Immunization of diffusion models against malicious Fine-Tuning with safe concepts retention](https://mlsanity.com/paper/2507.13598/) (2025)
- [Interactionless Inverse Reinforcement Learning: A Data-Centric Framework for Durable Alignment](https://mlsanity.com/paper/2602.14844/) (2026)
- [Unlearning the Unpromptable: Prompt-free Instance Unlearning in Diffusion Models](https://mlsanity.com/paper/2603.10445/) (2026)
- [Reference-Guided Machine Unlearning](https://mlsanity.com/paper/2603.11210/) (2026)
- [Unleashing Uncertainty: Efficient Machine Unlearning for Generative AI](https://mlsanity.com/paper/2508.20773/) (2025)
- [Lethe: Principled Dual-Stream Update for Persistent Knowledge Erasure in Federated Unlearning](https://mlsanity.com/paper/2601.22601/) (2026)
- [Retain-Neutral Surrogates for Min-Max Unlearning](https://mlsanity.com/paper/2605.05871/) (2026)
- [ICED: Concept-level Machine Unlearning via Interpretable Concept Decomposition](https://mlsanity.com/paper/2605.14309/) (2026)
- [BARRIER: Bounded Activation Regions for Robust Information Erasure](https://mlsanity.com/paper/2605.15737/) (2026)
- [On the Hidden Costs of Counterfactual Knowledge Training in LLM Unlearning](https://mlsanity.com/paper/2605.27083/) (2026)
- [Heterogeneous Knowledge Distillation via Geometry Decoupling and Momentum-Aware Gradient Regulation](https://mlsanity.com/paper/2606.24557/) (2026)
- [TILDE: TILt-based Distributional Erasure for Concept Unlearning](https://mlsanity.com/paper/2607.06432/) (2026)
- [Spectral Saliency for Machine Unlearning](https://mlsanity.com/paper/2608.15548/) (2026)

## Abstract (arXiv)

Machine Unlearning has recently garnered significant attention, aiming to selectively remove knowledge associated with specific data while preserving the model's performance on the remaining data. A fundamental challenge in this process is balancing effective unlearning with knowledge retention, as naive optimization of these competing objectives can lead to conflicting gradients, hindering convergence and degrading overall performance. To address this issue, we propose Learning to Unlearn while Retaining, aimed to mitigate gradient conflicts between unlearning and retention objectives. Our approach strategically avoids conflicts through an implicit gradient regularization mechanism that emerges naturally within the proposed framework. This prevents conflicting gradients between unlearning and retention, leading to effective unlearning while preserving the model's utility. We validate our approach across both discriminative and generative tasks, demonstrating its effectiveness in achieving unlearning without compromising performance on remaining data. Our results highlight the advantages of avoiding such gradient conflicts, outperforming existing methods that fail to account for these interactions.