# Class Unlearning via Depth-Aware Removal of Forget-Specific Directions

arXiv: https://arxiv.org/abs/2604.15166 · HTML: https://mlsanity.com/paper/2604.15166/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Machine unlearning, one of 12 directions in LLM safety and security.
- Builds on 30 of its 37 references that are in the map.
- First posted on arXiv 16 Apr 2026.

## Placed in

- LLM safety and security / [Machine unlearning](https://mlsanity.com/safety/machine-unlearning/)

## Builds on (most cited first, 30 of 30)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [How transferable are features in deep neural networks?](https://mlsanity.com/paper/1411.1792/) (2014)
- [Machine Unlearning](https://mlsanity.com/paper/1912.03817/) (2019)
- [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022)
- [SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation](https://mlsanity.com/paper/2310.12508/) (2023)
- [Towards Unbounded Machine Unlearning](https://mlsanity.com/paper/2302.09880/) (2023)
- [Model Sparsity Can Simplify Machine Unlearning](https://mlsanity.com/paper/2304.04934/) (2023)
- [Fast Machine Unlearning Without Retraining Through Selective Synaptic Dampening](https://mlsanity.com/paper/2308.07707/) (2023)
- [Zero-Shot Machine Unlearning](https://mlsanity.com/paper/2201.05629/) (2022)
- [Machine Unlearning: Linear Filtration for Logit-based Classifiers](https://mlsanity.com/paper/2002.02730/) (2020)
- [Inexact Unlearning Needs More Careful Evaluations to Avoid a False Sense of Privacy](https://mlsanity.com/paper/2403.01218/) (2024)
- [Forget Unlearning: Towards True Data-Deletion in Machine Learning](https://mlsanity.com/paper/2210.08911/) (2022)
- [Unified Gradient-Based Machine Unlearning with Remain Geometry Enhancement](https://mlsanity.com/paper/2409.19732/) (2024)
- [Machine Unlearning: A Comprehensive Survey](https://mlsanity.com/paper/2405.07406/) (2024)
- [Model Editing as a Robust and Denoised variant of DPO: A Case Study on Toxicity](https://mlsanity.com/paper/2405.13967/) (2024)
- [Decoupled Distillation to Erase: A General Unlearning Method for Any Class-centric Tasks](https://mlsanity.com/paper/2503.23751/) (2025)
- [CURE: Concept Unlearning via Orthogonal Representation Editing in Diffusion Models](https://mlsanity.com/paper/2505.12677/) (2025)
- [Deep Unlearning: Fast and Efficient Gradient-free Approach to Class Forgetting](https://mlsanity.com/paper/2312.00761/) (2023)
- [Is Retain Set All You Need in Machine Unlearning? Restoring Performance of Unlearned Models with Out-Of-Distribution Images](https://mlsanity.com/paper/2404.12922/) (2024)
- [Decoupling the Class Label and the Target Concept in Machine Unlearning](https://mlsanity.com/paper/2406.08288/) (2024)
- [Ascent Fails to Forget](https://mlsanity.com/paper/2509.26427/) (2025)
- [Towards Natural Machine Unlearning](https://mlsanity.com/paper/2405.15495/) (2024)
- [Deep Unlearn: Benchmarking Machine Unlearning for Image Classification](https://mlsanity.com/paper/2410.01276/) (2024)
- [O-Edit: Orthogonal Subspace Editing for Language Model Sequential Editing](https://mlsanity.com/paper/2410.11469/) (2024)
- [Random Relabeling for Efficient Machine Unlearning](https://mlsanity.com/paper/2305.12320/) (2023)

## Built on by (most cited first, 4 of 4)

- [Classification-Head Bias in Class-Level Machine Unlearning: Diagnosis, Mitigation, and Evaluation](https://mlsanity.com/paper/2605.08730/) (2026)
- [InstantForget: Update-Free Backdoor Unlearning with Inference-Time Feature Reset](https://mlsanity.com/paper/2606.15730/) (2026)
- [Gradient Concentration, Not Weight Saliency, Explains Representation-Level Class Unlearning](https://mlsanity.com/paper/2607.21353/) (2026)
- [Unmerge: Efficient Machine Unlearning via Task Arithmetic](https://mlsanity.com/paper/2609.38895/) (2026)

## Abstract (arXiv)

Machine unlearning aims to remove targeted knowledge from a trained model without the cost of retraining from scratch. In class unlearning, however, reducing accuracy on forget classes does not necessarily imply true forgetting: forgotten information can remain encoded in internal representations, and apparent forgetting may arise from classifier-head suppression rather than representational removal. We show that existing class-unlearning methods often exhibit weak or negative selectivity, preserve forget-class structure in deep representations, or rely heavily on final-layer bias shifts. We then introduce DAMP (Depth-Aware Modulation by Projection), a one-shot, closed-form weight-surgery method that removes forget-specific directions from a pretrained network without gradient-based optimization. At each stage, DAMP computes class prototypes in the input space of the next learnable operator, extracts forget directions as residuals relative to retain-class prototypes, and applies a projection-based update to reduce downstream sensitivity to those directions. To preserve utility, DAMP uses a parameter-free depth-aware scaling rule derived from probe separability, applying smaller edits in early layers and larger edits in deeper layers. The method naturally extends to multi-class forgetting through low-rank subspace removal. Across MNIST, CIFAR-10, CIFAR-100, and Tiny ImageNet, and across convolutional and transformer architectures, DAMP more closely resembles the retraining gold standard than some of the prior methods, improving selective forgetting while better preserving retain-class performance and reducing residual forget-class structure in deep layers.