# Distributional Machine Unlearning via Selective Data Removal

arXiv: https://arxiv.org/abs/2507.15112 · HTML: https://mlsanity.com/paper/2507.15112/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Machine unlearning, one of 12 directions in LLM safety and security.
- Builds on 31 of its 40 references that are in the map.
- First posted on arXiv 20 Jul 2025.

## Placed in

- LLM safety and security / [Machine unlearning](https://mlsanity.com/safety/machine-unlearning/)

## Builds on (most cited first, 30 of 31)

- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Machine Unlearning](https://mlsanity.com/paper/1912.03817/) (2019)
- [Certified Data Removal from Machine Learning Models](https://mlsanity.com/paper/1911.03030/) (2019)
- [Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection](https://mlsanity.com/paper/2004.07667/) (2020)
- [Learning the Difference that Makes a Difference with Counterfactually-Augmented Data](https://mlsanity.com/paper/1909.12434/) (2019)
- [SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation](https://mlsanity.com/paper/2310.12508/) (2023)
- [Who's Harry Potter? Approximate Unlearning in LLMs](https://mlsanity.com/paper/2310.02238/) (2023)
- [Descent-to-Delete: Gradient-Based Methods for Machine Unlearning](https://mlsanity.com/paper/2007.02923/) (2020)
- [From Quantity to Quality: Boosting LLM Performance with Self-Guided Data Selection for Instruction Tuning](https://mlsanity.com/paper/2308.12032/) (2023)
- [GRAD-MATCH: Gradient Matching based Data Subset Selection for Efficient Deep Model Training](https://mlsanity.com/paper/2103.00123/) (2021)
- [Towards Unbounded Machine Unlearning](https://mlsanity.com/paper/2302.09880/) (2023)
- [LEACE: Perfect linear concept erasure in closed form](https://mlsanity.com/paper/2306.03819/) (2023)
- [Approximate Data Deletion from Machine Learning Models](https://mlsanity.com/paper/2002.10077/) (2020)
- [Fast Yet Effective Machine Unlearning](https://mlsanity.com/paper/2111.08947/) (2021)
- [Adversarial Removal of Demographic Attributes from Text Data](https://mlsanity.com/paper/1808.06640/) (2018)
- [Autoencoding Variational Inference For Topic Models](https://mlsanity.com/paper/1703.01488/) (2017)
- [Coverage-centric Coreset Selection for High Pruning Rates](https://mlsanity.com/paper/2210.15809/) (2022)
- [A Survey on Data Selection for LLM Instruction Tuning](https://mlsanity.com/paper/2402.05123/) (2024)
- [Forget Unlearning: Towards True Data-Deletion in Machine Learning](https://mlsanity.com/paper/2210.08911/) (2022)
- [Langevin Unlearning: A New Perspective of Noisy Gradient Descent for Machine Unlearning](https://mlsanity.com/paper/2401.10371/) (2024)
- [One-Shot Learning as Instruction Data Prospector for Large Language Models](https://mlsanity.com/paper/2312.10302/) (2023)
- [Spanning Training Progress: Temporal Dual-Depth Scoring (TDDS) for Enhanced Dataset Pruning](https://mlsanity.com/paper/2311.13613/) (2023)
- [Towards Certified Unlearning for Deep Neural Networks](https://mlsanity.com/paper/2408.00920/) (2024)
- [Certified Unlearning for Neural Networks](https://mlsanity.com/paper/2506.06985/) (2025)
- [Deep Unlearning: Fast and Efficient Gradient-free Approach to Class Forgetting](https://mlsanity.com/paper/2312.00761/) (2023)
- [Language Models May Verbatim Complete Text They Were Not Explicitly Trained On](https://mlsanity.com/paper/2503.17514/) (2025)
- [The Utility and Complexity of in- and out-of-Distribution Machine Unlearning](https://mlsanity.com/paper/2412.09119/) (2024)
- [Data Whisperer: Efficient Data Selection for Task-Specific LLM Fine-Tuning via Few-Shot In-Context Learning](https://mlsanity.com/paper/2505.12212/) (2025)
- [Fast Rates in Pool-Based Batch Active Learning](https://mlsanity.com/paper/2202.05448/) (2022)

## Built on by (most cited first, 5 of 5)

- [Statistical Unlearning of Distributions: A Hypothesis Testing Approach](https://mlsanity.com/paper/2605.16645/) (2026)
- [SoK: Unlearnability and Unlearning for Model Dememorization](https://mlsanity.com/paper/2605.11592/) (2026)
- [What to Forget in Unlearning? Forget Set Curation for Language Models](https://mlsanity.com/paper/2608.14855/) (2026)
- [Bounding Retraining Equivalence and the Deletion Floor in Materials Machine Unlearning](https://mlsanity.com/paper/2609.35635/) (2026)
- [Learning What to Forget: Distributional Unlearning for LLM Representation Spaces](https://mlsanity.com/paper/2609.38929/) (2026)

## Abstract (arXiv)

Machine learning systems increasingly face requirements to remove entire domains of information--such as toxic language or biases--rather than individual user data. This task presents a dilemma: full removal of the unwanted domain data is computationally expensive, while random partial removal is statistically inefficient. We find that a domain's statistical influence is often concentrated in a small subset of its data samples, suggesting a path between ineffective partial removal and unnecessary complete removal. We formalize this as distributional unlearning: a framework to select a small subset that balances forgetting an unwanted distribution while preserving a desired one. Using Kullback-Leibler divergence constraints, we derive the exact removal-preservation Pareto frontier for Gaussian distributions and prove that models trained on the edited data achieve corresponding log-loss bounds. We propose a distance-based selection algorithm and show it is quadratically more sample-efficient than random removal in the challenging low-divergence regime. Experiments across synthetic, text, and image datasets (Jigsaw, CIFAR-10, SMS spam) show our method requires 15-82% less deletion than full removal for strong unlearning effects, e.g., halving initial forget set accuracy. Ultimately, by showing a small forget set often suffices, our framework lays the foundations for more scalable and rigorous subpopulation unlearning.