# Learning to Unlearn: Instance-wise Unlearning for Pre-trained Classifiers

arXiv: https://arxiv.org/abs/2301.11578 · HTML: https://mlsanity.com/paper/2301.11578/

## Key facts

- Cited by 67 papers in the map (data through 14 Oct 2026).
- Placed in Machine unlearning, one of 12 directions in LLM safety and security.
- Builds on 47 of its 60 references that are in the map.
- First posted on arXiv 27 Jan 2023.

## Placed in

- LLM safety and security / [Machine unlearning](https://mlsanity.com/safety/machine-unlearning/)

## Builds on (most cited first, 30 of 47)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- [Intriguing properties of neural networks](https://mlsanity.com/paper/1312.6199/) (2013)
- [MobileNetV2: Inverted Residuals and Linear Bottlenecks](https://mlsanity.com/paper/1801.04381/) (2018)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Adversarial examples in the physical world](https://mlsanity.com/paper/1607.02533/) (2016)
- [Similarity of Neural Network Representations Revisited](https://mlsanity.com/paper/1905.00414/) (2019)
- [Natural Adversarial Examples](https://mlsanity.com/paper/1907.07174/) (2019)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model size](https://mlsanity.com/paper/1602.07360/) (2016)
- [Adversarial Examples Are Not Bugs, They Are Features](https://mlsanity.com/paper/1905.02175/) (2019)
- [Riemannian Walk for Incremental Learning: Understanding Forgetting and Intransigence](https://mlsanity.com/paper/1801.10112/) (2018)
- [An Empirical Study of Example Forgetting during Deep Neural Network Learning](https://mlsanity.com/paper/1812.05159/) (2018)
- [Learning Efficient Convolutional Networks through Network Slimming](https://mlsanity.com/paper/1708.06519/) (2017)
- [Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks](https://mlsanity.com/paper/1911.04933/) (2019)
- [Learning Structured Sparsity in Deep Neural Networks](https://mlsanity.com/paper/1608.03665/) (2016)
- [Importance Estimation for Neural Network Pruning](https://mlsanity.com/paper/1906.10771/) (2019)
- [ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Models](https://mlsanity.com/paper/1806.01246/) (2018)
- [Making AI Forget You: Data Deletion in Machine Learning](https://mlsanity.com/paper/1907.05012/) (2019)
- [Age Progression/Regression by Conditional Adversarial Autoencoder](https://mlsanity.com/paper/1702.08423/) (2017)
- [Label-Only Membership Inference Attacks](https://mlsanity.com/paper/2007.14321/) (2020)
- [Amnesiac Machine Learning](https://mlsanity.com/paper/2010.10981/) (2020)
- [White-box vs Black-box: Bayes Optimal Strategies for Membership Inference](https://mlsanity.com/paper/1908.11229/) (2019)

## Built on by (most cited first, 30 of 67)

- [Forget-Me-Not: Learning to Forget in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.17591/) (2023)
- [Machine Unlearning of Pre-trained Large Language Models](https://mlsanity.com/paper/2402.15159/) (2024)
- [Exploring the Landscape of Machine Unlearning: A Comprehensive Survey and Taxonomy](https://mlsanity.com/paper/2305.06360/) (2023)
- [Single Image Unlearning: Efficient Machine Unlearning in Multimodal Large Language Models](https://mlsanity.com/paper/2405.12523/) (2024)
- [Towards Robust and Parameter-Efficient Knowledge Unlearning for LLMs](https://mlsanity.com/paper/2408.06621/) (2024)
- [Machine Unlearning: A Comprehensive Survey](https://mlsanity.com/paper/2405.07406/) (2024)
- [On Effects of Steering Latent Representation for Large Language Model Unlearning](https://mlsanity.com/paper/2408.06223/) (2024)
- [Continual Forgetting for Pre-trained Vision Models](https://mlsanity.com/paper/2403.11530/) (2024)
- [Decoupled Distillation to Erase: A General Unlearning Method for Any Class-centric Tasks](https://mlsanity.com/paper/2503.23751/) (2025)
- [To Forget or Not? Towards Practical Knowledge Unlearning for Large Language Models](https://mlsanity.com/paper/2407.01920/) (2024)
- [Unified Parameter-Efficient Unlearning for LLMs](https://mlsanity.com/paper/2412.00383/) (2024)
- [Learning to Unlearn for Robust Machine Unlearning](https://mlsanity.com/paper/2407.10494/) (2024)
- [Are We Truly Forgetting? A Critical Re-examination of Machine Unlearning Evaluation Protocols](https://mlsanity.com/paper/2503.06991/) (2025)
- [MU-Bench: A Multitask Multimodal Benchmark for Machine Unlearning](https://mlsanity.com/paper/2406.14796/) (2024)
- [MUNBa: Machine Unlearning via Nash Bargaining](https://mlsanity.com/paper/2411.15537/) (2024)
- [Contrastive Unlearning: A Contrastive Approach to Machine Unlearning](https://mlsanity.com/paper/2401.10458/) (2024)
- [A Survey on Machine Unlearning: Techniques and New Emerged Privacy Risks](https://mlsanity.com/paper/2406.06186/) (2024)
- [LoTUS: Large-Scale Machine Unlearning with a Taste of Uncertainty](https://mlsanity.com/paper/2503.18314/) (2025)
- [Layer Attack Unlearning: Fast and Accurate Machine Unlearning via Layer Level Attack and Knowledge Distillation](https://mlsanity.com/paper/2312.16823/) (2023)
- [Towards Natural Machine Unlearning](https://mlsanity.com/paper/2405.15495/) (2024)
- [Revisiting Machine Unlearning with Dimensional Alignment](https://mlsanity.com/paper/2407.17710/) (2024)
- [The Landscape of Memorization in LLMs: Mechanisms, Measurement, and Mitigation](https://mlsanity.com/paper/2507.05578/) (2025)
- [Label Smoothing Improves Machine Unlearning](https://mlsanity.com/paper/2406.07698/) (2024)
- [A Certified Unlearning Approach without Access to Source Data](https://mlsanity.com/paper/2506.06486/) (2025)
- [Fast-FedUL: A Training-Free Federated Unlearning with Provable Skew Resilience](https://mlsanity.com/paper/2405.18040/) (2024)
- [Distribution-Level Feature Distancing for Machine Unlearning: Towards a Better Trade-off Between Model Utility and Forgetting](https://mlsanity.com/paper/2409.14747/) (2024)
- [A Unified Gradient-based Framework for Task-agnostic Continual Learning-Unlearning](https://mlsanity.com/paper/2505.15178/) (2025)
- [Towards Source-Free Machine Unlearning](https://mlsanity.com/paper/2508.15127/) (2025)
- [POUR: A Provably Optimal Method for Unlearning Representations via Neural Collapse](https://mlsanity.com/paper/2511.19339/) (2025)
- [Practical Continual Forgetting for Pre-trained Vision Models](https://mlsanity.com/paper/2501.09705/) (2025)

## Abstract (arXiv)

Since the recent advent of regulations for data protection (e.g., the General Data Protection Regulation), there has been increasing demand in deleting information learned from sensitive data in pre-trained models without retraining from scratch. The inherent vulnerability of neural networks towards adversarial attacks and unfairness also calls for a robust method to remove or correct information in an instance-wise fashion, while retaining the predictive performance across remaining data. To this end, we consider instance-wise unlearning, of which the goal is to delete information on a set of instances from a pre-trained model, by either misclassifying each instance away from its original prediction or relabeling the instance to a different label. We also propose two methods that reduce forgetting on the remaining data: 1) utilizing adversarial examples to overcome forgetting at the representation-level and 2) leveraging weight importance metrics to pinpoint network parameters guilty of propagating unwanted information. Both methods only require the pre-trained model and data instances to forget, allowing painless application to real-life settings where the entire training set is unavailable. Through extensive experimentation on various image classification benchmarks, we show that our approach effectively preserves knowledge of remaining data while unlearning given instances in both single-task and continual unlearning scenarios.