# Deep Unlearning: Fast and Efficient Gradient-free Approach to Class Forgetting

arXiv: https://arxiv.org/abs/2312.00761 · HTML: https://mlsanity.com/paper/2312.00761/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Placed in Machine unlearning, one of 12 directions in LLM safety and security.
- Builds on 30 of its 48 references that are in the map.
- First posted on arXiv 1 Dec 2023.

## Placed in

- LLM safety and security / [Machine unlearning](https://mlsanity.com/safety/machine-unlearning/)

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Continual Lifelong Learning with Neural Networks: A Review](https://mlsanity.com/paper/1802.07569/) (2018)
- [Machine Unlearning](https://mlsanity.com/paper/1912.03817/) (2019)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [Deep Learning and the Information Bottleneck Principle](https://mlsanity.com/paper/1503.02406/) (2015)
- [Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks](https://mlsanity.com/paper/1911.04933/) (2019)
- [Opening the Black Box of Deep Neural Networks via Information](https://mlsanity.com/paper/1703.00810/) (2017)
- [Making AI Forget You: Data Deletion in Machine Learning](https://mlsanity.com/paper/1907.05012/) (2019)
- [Gradient Projection Memory for Continual Learning](https://mlsanity.com/paper/2103.09762/) (2021)
- [Towards Unbounded Machine Unlearning](https://mlsanity.com/paper/2302.09880/) (2023)
- [Approximate Data Deletion from Machine Learning Models](https://mlsanity.com/paper/2002.10077/) (2020)
- [Machine Unlearning of Features and Labels](https://mlsanity.com/paper/2108.11577/) (2021)
- [Fast Yet Effective Machine Unlearning](https://mlsanity.com/paper/2111.08947/) (2021)
- [Fast Machine Unlearning Without Retraining Through Selective Synaptic Dampening](https://mlsanity.com/paper/2308.07707/) (2023)
- [Forgetting Outside the Box: Scrubbing Deep Networks of Information Accessible from Input-Output Observations](https://mlsanity.com/paper/2003.02960/) (2020)
- [Mixed-Privacy Forgetting in Deep Networks](https://mlsanity.com/paper/2012.13431/) (2020)
- [Zero-Shot Machine Unlearning](https://mlsanity.com/paper/2201.05629/) (2022)
- [Machine Unlearning for Random Forests](https://mlsanity.com/paper/2009.05567/) (2020)
- [Variational Bayesian Unlearning](https://mlsanity.com/paper/2010.12883/) (2020)
- [Machine Unlearning: Linear Filtration for Logit-based Classifiers](https://mlsanity.com/paper/2002.02730/) (2020)
- [Towards Adversarial Evaluations for Inexact Machine Unlearning](https://mlsanity.com/paper/2201.06640/) (2022)
- [Certifiable Machine Unlearning for Linear Models](https://mlsanity.com/paper/2106.15093/) (2021)
- [Continual Learning with Scaled Gradient Projection](https://mlsanity.com/paper/2302.01386/) (2023)
- [Few-Shot Unlearning by Model Inversion](https://mlsanity.com/paper/2205.15567/) (2022)
- [On Training Sample Memorization: Lessons from Benchmarking Generative Modeling with a Large-scale Competition](https://mlsanity.com/paper/2106.03062/) (2021)
- [Subspace based Federated Unlearning](https://mlsanity.com/paper/2302.12448/) (2023)
- [Class Gradient Projection For Continual Learning](https://mlsanity.com/paper/2311.14905/) (2023)

## Built on by (most cited first, 22 of 22)

- [SAP: Corrective Machine Unlearning with Scaled Activation Projection for Label Noise Robustness](https://mlsanity.com/paper/2403.08618/) (2024)
- [ESC: Erasing Space Concept for Knowledge Deletion](https://mlsanity.com/paper/2504.02199/) (2025)
- [Forget Vectors at Play: Universal Input Perturbations Driving Machine Unlearning in Image Classification](https://mlsanity.com/paper/2412.16780/) (2024)
- [POUR: A Provably Optimal Method for Unlearning Representations via Neural Collapse](https://mlsanity.com/paper/2511.19339/) (2025)
- [Distributional Machine Unlearning via Selective Data Removal](https://mlsanity.com/paper/2507.15112/) (2025)
- [Geometric-disentangelment Unlearning](https://mlsanity.com/paper/2511.17100/) (2025)
- [The Unseen Threat: Residual Knowledge in Machine Unlearning under Perturbed Samples](https://mlsanity.com/paper/2601.22359/) (2026)
- [An Illusion of Unlearning? Assessing Machine Unlearning Through Internal Representations](https://mlsanity.com/paper/2604.08271/) (2026)
- [Class Unlearning via Depth-Aware Removal of Forget-Specific Directions](https://mlsanity.com/paper/2604.15166/) (2026)
- [On the Necessity of Output Distribution Reweighting for Effective Class Unlearning](https://mlsanity.com/paper/2506.20893/) (2025)
- [Direct Token Optimization: A Self-contained Approach to Large Language Model Unlearning](https://mlsanity.com/paper/2510.00125/) (2025)
- [Representation Unlearning: Forgetting through Information Compression](https://mlsanity.com/paper/2601.21564/) (2026)
- [FactorHD: A Hyperdimensional Computing Model for Multi-Object Multi-Class Representation and Factorization](https://mlsanity.com/paper/2507.12366/) (2025)
- [Exploring and Leveraging Class Vectors for Classifier Editing](https://mlsanity.com/paper/2510.11268/) (2025)
- [ModHiFi: Identifying High Fidelity predictive components for Model Modification](https://mlsanity.com/paper/2511.19566/) (2025)
- [Statistical Unlearning of Distributions: A Hypothesis Testing Approach](https://mlsanity.com/paper/2605.16645/) (2026)
- [Erased, but Not Gone: Output Forgetting Is Not True Forgetting](https://mlsanity.com/paper/2606.25001/) (2026)
- [Mirror Mirror on the Wall, Have I Forgotten it All? A New Framework for Evaluating Machine Unlearning](https://mlsanity.com/paper/2505.08138/) (2025)
- [Memory Undone: Between Knowing and Not Knowing in Data Systems](https://mlsanity.com/paper/2602.21180/) (2026)
- [SCOPE: Entanglement Frontier Escape for Source-Free Class Unlearning](https://mlsanity.com/paper/2608.02058/) (2026)
- [Open Vocabulary Domain Unlearning](https://mlsanity.com/paper/2609.31356/) (2026)
- [Unmerge: Efficient Machine Unlearning via Task Arithmetic](https://mlsanity.com/paper/2609.38895/) (2026)

## Abstract (arXiv)

Machine unlearning is a prominent and challenging field, driven by regulatory demands for user data deletion and heightened privacy awareness. Existing approaches involve retraining model or multiple finetuning steps for each deletion request, often constrained by computational limits and restricted data access. In this work, we introduce a novel class unlearning algorithm designed to strategically eliminate specific classes from the learned model. Our algorithm first estimates the Retain and the Forget Spaces using Singular Value Decomposition on the layerwise activations for a small subset of samples from the retain and unlearn classes, respectively. We then compute the shared information between these spaces and remove it from the forget space to isolate class-discriminatory feature space. Finally, we obtain the unlearned model by updating the weights to suppress the class discriminatory features from the activation spaces. We demonstrate our algorithm's efficacy on ImageNet using a Vision Transformer with only ~ 1.5% drop in retain accuracy compared to the original model while maintaining under 1% accuracy on the unlearned class samples. Furthermore, our algorithm exhibits competitive unlearning performance and resilience against Membership Inference Attacks (MIA). Compared to baselines, it achieves an average accuracy improvement of 1.38% on the ImageNet dataset while requiring up to 10 × fewer samples for unlearning. Additionally, under stronger MIA attacks on the CIFAR-100 dataset using a ResNet18 architecture, our approach outperforms the best baseline by 1.8%. Our code is available at https://github.com/sangamesh-kodge/classforgetting.