# Data Selection for Transfer Unlearning arXiv: https://arxiv.org/abs/2405.10425 ยท HTML: https://mlsanity.com/paper/2405.10425/ ## Key facts - Cited by 5 papers in the map (data through 14 Oct 2026). - Placed in Machine unlearning, one of 12 directions in LLM safety and security. - Builds on 29 of its 43 references that are in the map. - First posted on arXiv 16 May 2024. ## Placed in - LLM safety and security / [Machine unlearning](https://mlsanity.com/safety/machine-unlearning/) ## Builds on (most cited first, 29 of 29) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019) - [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014) - [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019) - [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013) - [Machine Unlearning](https://mlsanity.com/paper/1912.03817/) (2019) - [Remote Sensing Image Scene Classification: Benchmark and State of the Art](https://mlsanity.com/paper/1703.00121/) (2017) - [Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks](https://mlsanity.com/paper/1911.04933/) (2019) - [Learning multiple visual domains with residual adapters](https://mlsanity.com/paper/1705.08045/) (2017) - [Making AI Forget You: Data Deletion in Machine Learning](https://mlsanity.com/paper/1907.05012/) (2019) - [Remember What You Want to Forget: Algorithms for Machine Unlearning](https://mlsanity.com/paper/2103.03279/) (2021) - [A Comprehensive Survey on Pretrained Foundation Models: A History from BERT to ChatGPT](https://mlsanity.com/paper/2302.09419/) (2023) - [Amnesiac Machine Learning](https://mlsanity.com/paper/2010.10981/) (2020) - [A Survey of Machine Unlearning](https://mlsanity.com/paper/2209.02299/) (2022) - [Large Language Model Unlearning](https://mlsanity.com/paper/2310.10683/) (2023) - [Unrolling SGD: Understanding Factors Influencing Machine Unlearning](https://mlsanity.com/paper/2109.13398/) (2021) - [Towards Unbounded Machine Unlearning](https://mlsanity.com/paper/2302.09880/) (2023) - [In-Context Unlearning: Language Models as Few Shot Unlearners](https://mlsanity.com/paper/2310.07579/) (2023) - [When Machine Unlearning Jeopardizes Privacy](https://mlsanity.com/paper/2005.02205/) (2020) - [Forgetting Outside the Box: Scrubbing Deep Networks of Information Accessible from Input-Output Observations](https://mlsanity.com/paper/2003.02960/) (2020) - [SpotTune: Transfer Learning through Adaptive Fine-tuning](https://mlsanity.com/paper/1811.08737/) (2018) - [Mixed-Privacy Forgetting in Deep Networks](https://mlsanity.com/paper/2012.13431/) (2020) - [Towards Adversarial Evaluations for Inexact Machine Unlearning](https://mlsanity.com/paper/2201.06640/) (2022) - [Head2Toe: Utilizing Intermediate Representations for Better Transfer Learning](https://mlsanity.com/paper/2201.03529/) (2022) - [Inexact Unlearning Needs More Careful Evaluations to Avoid a False Sense of Privacy](https://mlsanity.com/paper/2403.01218/) (2024) - [Scalable Transfer Learning with Expert Models](https://mlsanity.com/paper/2009.13239/) (2020) - [Identifying Spurious Biases Early in Training through the Lens of Simplicity Bias](https://mlsanity.com/paper/2305.18761/) (2023) - [Improved Fine-Tuning by Better Leveraging Pre-Training Data](https://mlsanity.com/paper/2111.12292/) (2021) - [Weighted Training for Cross-Task Learning](https://mlsanity.com/paper/2105.14095/) (2021) ## Built on by (most cited first, 5 of 5) - [Large Language Model Unlearning via Embedding-Corrupted Prompts](https://mlsanity.com/paper/2406.07933/) (2024) - [Towards flexible perception with visual memory](https://mlsanity.com/paper/2408.08172/) (2024) - [Parallel Unlearning in Inherited Model Networks](https://mlsanity.com/paper/2408.08493/) (2024) - [Unsupervised Data Validation Methods for Efficient Model Training](https://mlsanity.com/paper/2410.07880/) (2024) - [Stochastic Meta-Unlearning: Bridging Language Backbone and Multimodal Unlearning](https://mlsanity.com/paper/2607.18615/) (2026) ## Abstract (arXiv) As deep learning models are becoming larger and data-hungrier, there are growing ethical, legal and technical concerns over use of data: in practice, agreements on data use may change over time, rendering previously-used training data impermissible for training purposes. These issues have driven increased attention to machine unlearning: removing "the influence of" a subset of training data from a trained model. In this work, we advocate for a relaxed definition of unlearning that does not address privacy applications but targets a scenario where a data owner withdraws permission of use of their data for training purposes. In this context, we consider the important problem of transfer unlearning where a pretrained model is transferred to a target dataset that contains some "non-static" data that may need to be unlearned in the future. We propose a new method that uses a mechanism for selecting relevant examples from an auxiliary "static" dataset, and finetunes on the selected data instead of "non-static" target data; addressing all unlearning requests ahead of time. We also adapt a recent relaxed definition of unlearning to our problem setting and demonstrate that our approach is an exact transfer unlearner according to it, while being highly efficient (amortized). We find that our method outperforms the gold standard "exact unlearning" (finetuning on only the "static" portion of the target dataset) on several datasets, especially for small "static" sets, sometimes approaching an upper bound for test accuracy. We also analyze factors influencing the accuracy boost obtained by data selection.