# Learn to Unlearn for Deep Neural Networks: Minimizing Unlearning Interference with Gradient Projection

arXiv: https://arxiv.org/abs/2312.04095 · HTML: https://mlsanity.com/paper/2312.04095/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Placed in Machine unlearning, one of 12 directions in LLM safety and security.
- Builds on 31 of its 48 references that are in the map.
- First posted on arXiv 7 Dec 2023.

## Placed in

- LLM safety and security / [Machine unlearning](https://mlsanity.com/safety/machine-unlearning/)

## Builds on (most cited first, 30 of 31)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [Machine Unlearning](https://mlsanity.com/paper/1912.03817/) (2019)
- [Striving for Simplicity: The All Convolutional Net](https://mlsanity.com/paper/1412.6806/) (2014)
- [Membership Inference Attacks From First Principles](https://mlsanity.com/paper/2112.03570/) (2021)
- [The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networks](https://mlsanity.com/paper/1802.08232/) (2018)
- [Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descent](https://mlsanity.com/paper/1902.06720/) (2019)
- [Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks](https://mlsanity.com/paper/1911.04933/) (2019)
- [Certified Data Removal from Machine Learning Models](https://mlsanity.com/paper/1911.03030/) (2019)
- [Exploiting Unintended Feature Leakage in Collaborative Learning](https://mlsanity.com/paper/1805.04049/) (2018)
- [Label-Only Membership Inference Attacks](https://mlsanity.com/paper/2007.14321/) (2020)
- [Remember What You Want to Forget: Algorithms for Machine Unlearning](https://mlsanity.com/paper/2103.03279/) (2021)
- [Gradient Projection Memory for Continual Learning](https://mlsanity.com/paper/2103.09762/) (2021)
- [White-box vs Black-box: Bayes Optimal Strategies for Membership Inference](https://mlsanity.com/paper/1908.11229/) (2019)
- [Approximate Data Deletion from Machine Learning Models](https://mlsanity.com/paper/2002.10077/) (2020)
- [Machine Unlearning of Features and Labels](https://mlsanity.com/paper/2108.11577/) (2021)
- [Differentially Private Learning with Adaptive Clipping](https://mlsanity.com/paper/1905.03871/) (2019)
- [Forgetting Outside the Box: Scrubbing Deep Networks of Information Accessible from Input-Output Observations](https://mlsanity.com/paper/2003.02960/) (2020)
- [Numerical Composition of Differential Privacy](https://mlsanity.com/paper/2106.02848/) (2021)
- [Mixed-Privacy Forgetting in Deep Networks](https://mlsanity.com/paper/2012.13431/) (2020)
- [Variational Bayesian Unlearning](https://mlsanity.com/paper/2010.12883/) (2020)
- [Machine Unlearning: Linear Filtration for Logit-based Classifiers](https://mlsanity.com/paper/2002.02730/) (2020)
- [Deep Unlearning via Randomized Conditionally Independent Hessians](https://mlsanity.com/paper/2204.07655/) (2022)
- [TRGP: Trust Region Gradient Projection for Continual Learning](https://mlsanity.com/paper/2202.02931/) (2022)
- [Mixed Differential Privacy in Computer Vision](https://mlsanity.com/paper/2203.11481/) (2022)
- [Markov Chain Monte Carlo-Based Machine Unlearning: Unlearning What Needs to be Forgotten](https://mlsanity.com/paper/2202.13585/) (2022)
- [Knowledge Removal in Sampling-based Bayesian Inference](https://mlsanity.com/paper/2203.12964/) (2022)

## Built on by (most cited first, 30 of 37)

- [Unified Gradient-Based Machine Unlearning with Remain Geometry Enhancement](https://mlsanity.com/paper/2409.19732/) (2024)
- [Learning to Unlearn while Retaining: Combating Gradient Conflicts in Machine Unlearning](https://mlsanity.com/paper/2503.06339/) (2025)
- [MU-Bench: A Multitask Multimodal Benchmark for Machine Unlearning](https://mlsanity.com/paper/2406.14796/) (2024)
- [Tool Unlearning for Tool-Augmented LLMs](https://mlsanity.com/paper/2502.01083/) (2025)
- [Remaining-data-free Machine Unlearning by Suppressing Sample Contribution](https://mlsanity.com/paper/2402.15109/) (2024)
- [Ferrari: Federated Feature Unlearning via Optimizing Feature Sensitivity](https://mlsanity.com/paper/2405.17462/) (2024)
- [ESC: Erasing Space Concept for Knowledge Deletion](https://mlsanity.com/paper/2504.02199/) (2025)
- [RAZOR: Ratio-Aware Layer Editing for Targeted Unlearning in Vision Transformers and Diffusion Models](https://mlsanity.com/paper/2603.14819/) (2026)
- [MUBox: A Critical Evaluation Framework of Deep Machine Unlearning](https://mlsanity.com/paper/2505.08576/) (2025)
- [A Unified Gradient-based Framework for Task-agnostic Continual Learning-Unlearning](https://mlsanity.com/paper/2505.15178/) (2025)
- [Towards Privacy-Guaranteed Label Unlearning in Vertical Federated Learning: Few-Shot Forgetting without Disclosure](https://mlsanity.com/paper/2410.10922/) (2024)
- [Efficient Utility-Preserving Machine Unlearning with Implicit Gradient Surgery](https://mlsanity.com/paper/2510.22124/) (2025)
- [Unlearning Personal Data from a Single Image](https://mlsanity.com/paper/2407.12069/) (2024)
- [Machine Unlearning under Overparameterization](https://mlsanity.com/paper/2505.22601/) (2025)
- [Reminiscence Attack on Residuals: Exploiting Approximate Machine Unlearning for Privacy](https://mlsanity.com/paper/2507.20573/) (2025)
- [Geometric-disentangelment Unlearning](https://mlsanity.com/paper/2511.17100/) (2025)
- [Group-robust Machine Unlearning](https://mlsanity.com/paper/2503.09330/) (2025)
- [Discriminative Adversarial Unlearning](https://mlsanity.com/paper/2402.06864/) (2024)
- [Machine Unlearning using Forgetting Neural Networks](https://mlsanity.com/paper/2410.22374/) (2024)
- [Exploring Incremental Unlearning: Techniques, Challenges, and Future Directions](https://mlsanity.com/paper/2502.16708/) (2025)
- [Ready2Unlearn: A Learning-Time Approach for Preparing Models with Future Unlearning Readiness](https://mlsanity.com/paper/2505.10845/) (2025)
- [Parallel Unlearning in Inherited Model Networks](https://mlsanity.com/paper/2408.08493/) (2024)
- [Neural Corrective Machine Unranking](https://mlsanity.com/paper/2411.08562/) (2024)
- [Understanding Machine Unlearning Through the Lens of Mode Connectivity](https://mlsanity.com/paper/2504.06407/) (2025)
- [Zero-Shot Machine Unlearning with Proxy Adversarial Data Generation](https://mlsanity.com/paper/2507.21738/) (2025)
- [WARP: Weight Teleportation for Attack-Resilient Unlearning Protocols](https://mlsanity.com/paper/2512.00272/) (2025)
- [Erased, but Not Gone: Output Forgetting Is Not True Forgetting](https://mlsanity.com/paper/2606.25001/) (2026)
- [The BatchNorm Illusion: Diagnosing Normalization Artifacts in Machine Unlearning Evaluation](https://mlsanity.com/paper/2609.08901/) (2026)
- [FedShard: Federated Unlearning with Efficiency Fairness and Performance Fairness](https://mlsanity.com/paper/2508.09866/) (2025)
- [Controllable Machine Unlearning via Gradient Pivoting](https://mlsanity.com/paper/2510.19226/) (2025)

## Abstract (arXiv)

Recent data-privacy laws have sparked interest in machine unlearning, which involves removing the effect of specific training samples from a learnt model as if they were never present in the original training dataset. The challenge of machine unlearning is to discard information about the ``forget'' data in the learnt model without altering the knowledge about the remaining dataset and to do so more efficiently than the naive retraining approach. To achieve this, we adopt a projected-gradient based learning method, named as Projected-Gradient Unlearning (PGU), in which the model takes steps in the orthogonal direction to the gradient subspaces deemed unimportant for the retaining dataset, so as to its knowledge is preserved. By utilizing Stochastic Gradient Descent (SGD) to update the model weights, our method can efficiently scale to any model and dataset size. We provide empirically evidence to demonstrate that our unlearning method can produce models that behave similar to models retrained from scratch across various metrics even when the training dataset is no longer accessible. Our code is available at https://github.com/hnanhtuan/projected_gradient_unlearning.