# Don't Forget Too Much: Towards Machine Unlearning on Feature Level

arXiv: https://arxiv.org/abs/2406.10951 · HTML: https://mlsanity.com/paper/2406.10951/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Placed in Machine unlearning, one of 12 directions in LLM safety and security.
- Builds on 22 of its 39 references that are in the map.
- First posted on arXiv 16 Jun 2024.

## Placed in

- LLM safety and security / [Machine unlearning](https://mlsanity.com/safety/machine-unlearning/)

## Builds on (most cited first, 22 of 22)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Machine Unlearning](https://mlsanity.com/paper/1912.03817/) (2019)
- [Striving for Simplicity: The All Convolutional Net](https://mlsanity.com/paper/1412.6806/) (2014)
- [Network Dissection: Quantifying Interpretability of Deep Visual Representations](https://mlsanity.com/paper/1704.05796/) (2017)
- [Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks](https://mlsanity.com/paper/1911.04933/) (2019)
- [Certified Data Removal from Machine Learning Models](https://mlsanity.com/paper/1911.03030/) (2019)
- [Amnesiac Machine Learning](https://mlsanity.com/paper/2010.10981/) (2020)
- [Detect What You Can: Detecting and Representing Objects using Holistic Models and Body Parts](https://mlsanity.com/paper/1406.2031/) (2014)
- [Machine Unlearning of Features and Labels](https://mlsanity.com/paper/2108.11577/) (2021)
- [Adaptive Machine Unlearning](https://mlsanity.com/paper/2106.04378/) (2021)
- [Graph Unlearning](https://mlsanity.com/paper/2103.14991/) (2021)
- [Zero-Shot Machine Unlearning](https://mlsanity.com/paper/2201.05629/) (2022)
- [Machine Unlearning for Random Forests](https://mlsanity.com/paper/2009.05567/) (2020)
- [Net2Vec: Quantifying and Explaining how Concepts are Encoded by Filters in Deep Neural Networks](https://mlsanity.com/paper/1801.03454/) (2018)
- [Variational Bayesian Unlearning](https://mlsanity.com/paper/2010.12883/) (2020)
- [Federated Unlearning via Class-Discriminative Pruning](https://mlsanity.com/paper/2110.11794/) (2021)
- [Machine Unlearning: Linear Filtration for Logit-based Classifiers](https://mlsanity.com/paper/2002.02730/) (2020)
- [Recommendation Unlearning](https://mlsanity.com/paper/2201.06820/) (2022)
- [Interpretable CNNs for Object Classification](https://mlsanity.com/paper/1901.02413/) (2019)
- [Interpretable Compositional Convolutional Neural Networks](https://mlsanity.com/paper/2107.04474/) (2021)
- [A Survey on Machine Unlearning: Techniques and New Emerged Privacy Risks](https://mlsanity.com/paper/2406.06186/) (2024)

## Built on by (most cited first, 11 of 11)

- [Machine Unlearning under Retain-Forget Entanglement](https://mlsanity.com/paper/2603.26569/) (2026)
- [Position: Bridge the Gaps between Machine Unlearning and AI Regulation](https://mlsanity.com/paper/2502.12430/) (2025)
- [Safe and Reliable Diffusion Models via Subspace Projection](https://mlsanity.com/paper/2503.16835/) (2025)
- [SUV: Scalable Large Language Model Copyright Compliance with Regularized Selective Unlearning](https://mlsanity.com/paper/2503.22948/) (2025)
- [Zero-shot Class Unlearning via Layer-wise Relevance Analysis and Neuronal Path Perturbation](https://mlsanity.com/paper/2410.23693/) (2024)
- [An Illusion of Unlearning? Assessing Machine Unlearning Through Internal Representations](https://mlsanity.com/paper/2604.08271/) (2026)
- [Game-Theoretic Machine Unlearning: Mitigating Extra Privacy Leakage](https://mlsanity.com/paper/2411.03914/) (2024)
- [Unlearning Algorithmic Biases over Graphs](https://mlsanity.com/paper/2505.14945/) (2025)
- [Bias-Aware Machine Unlearning: Towards Fairer Vision Models via Controllable Forgetting](https://mlsanity.com/paper/2509.07456/) (2025)
- [Forgetting Similar Samples: Can Machine Unlearning Do it Better?](https://mlsanity.com/paper/2601.06938/) (2026)
- [SAU: Sparsity-Aware Unlearning for LLMs via Gradient Masking and Importance Redistribution](https://mlsanity.com/paper/2602.00577/) (2026)

## Abstract (arXiv)

Machine unlearning enables pre-trained models to remove the effect of certain portions of training data. Previous machine unlearning schemes have mainly focused on unlearning a cluster of instances or all instances belonging to a specific class. These types of unlearning might have a significant impact on the model utility; and they may be inadequate for situations where we only need to unlearn features within instances, rather than the whole instances. Due to the different granularity, current unlearning methods can hardly achieve feature-level unlearning. To address the challenges of utility and granularity, we propose a refined granularity unlearning scheme referred to as ``feature unlearning". We first explore two distinct scenarios based on whether the annotation information about the features is given: feature unlearning with known annotations and feature unlearning without annotations. Regarding unlearning with known annotations, we propose an adversarial learning approach to automatically remove effects about features. For unlearning without annotations, we initially enable the output of one model's layer to identify different pattern features using model interpretability techniques. We proceed to filter features from instances based on these outputs with identifying ability. So that we can remove the feature impact based on filtered instances and the fine-tuning process. The effectiveness of our proposed approach is demonstrated through experiments involving diverse models on various datasets in different scenarios.