# Scalability of memorization-based machine unlearning

arXiv: https://arxiv.org/abs/2410.16516 · HTML: https://mlsanity.com/paper/2410.16516/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Machine unlearning, one of 12 directions in LLM safety and security.
- Builds on 17 of its 22 references that are in the map.
- First posted on arXiv 21 Oct 2024.

## Placed in

- LLM safety and security / [Machine unlearning](https://mlsanity.com/safety/machine-unlearning/)

## Builds on (most cited first, 17 of 17)

- [A Closer Look at Memorization in Deep Networks](https://mlsanity.com/paper/1706.05394/) (2017)
- [An Empirical Study of Example Forgetting during Deep Neural Network Learning](https://mlsanity.com/paper/1812.05159/) (2018)
- [Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks](https://mlsanity.com/paper/1911.04933/) (2019)
- [What Neural Networks Memorize and Why: Discovering the Long Tail via Influence Estimation](https://mlsanity.com/paper/2008.03703/) (2020)
- [Does Learning Require Memorization? A Short Tale about a Long Tail](https://mlsanity.com/paper/1906.05271/) (2019)
- [SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation](https://mlsanity.com/paper/2310.12508/) (2023)
- [A Survey of Machine Unlearning](https://mlsanity.com/paper/2209.02299/) (2022)
- [Machine Unlearning of Features and Labels](https://mlsanity.com/paper/2108.11577/) (2021)
- [Model Sparsity Can Simplify Machine Unlearning](https://mlsanity.com/paper/2304.04934/) (2023)
- [Robustness via curvature regularization, and vice versa](https://mlsanity.com/paper/1811.09716/) (2018)
- [The Privacy Onion Effect: Memorization is Relative](https://mlsanity.com/paper/2206.10469/) (2022)
- [Characterizing Structural Regularities of Labeled Data in Overparameterized Models](https://mlsanity.com/paper/2002.03206/) (2020)
- [What makes unlearning hard and what to do about it](https://mlsanity.com/paper/2406.01257/) (2024)
- [Are we making progress in unlearning? Findings from the first NeurIPS unlearning competition](https://mlsanity.com/paper/2406.09073/) (2024)
- [Distribution Density, Tails, and Outliers in Machine Learning: Metrics and Applications](https://mlsanity.com/paper/1910.13427/) (2019)
- [To Each (Textual Sequence) Its Own: Improving Memorized-Data Unlearning in Large Language Models](https://mlsanity.com/paper/2405.03097/) (2024)
- [Memorization Through the Lens of Curvature of Loss Function Around Samples](https://mlsanity.com/paper/2307.05831/) (2023)

## Built on by (most cited first, 5 of 5)

- [LLM Unlearning Reveals a Stronger-Than-Expected Coreset Effect in Current Benchmarks](https://mlsanity.com/paper/2504.10185/) (2025)
- [Leveraging Per-Instance Privacy for Machine Unlearning](https://mlsanity.com/paper/2505.18786/) (2025)
- [Lifting Data-Tracing Machine Unlearning to Knowledge-Tracing for Foundation Models](https://mlsanity.com/paper/2506.11253/) (2025)
- [Efficiently Attacking Memorization Scores](https://mlsanity.com/paper/2509.20463/) (2025)
- [Correlation-Guided Fast Machine Unlearning via Hessian Analysis](https://mlsanity.com/paper/2609.12620/) (2026)

## Abstract (arXiv)

Machine unlearning (MUL) focuses on removing the influence of specific subsets of data (such as noisy, poisoned, or privacy-sensitive data) from pretrained models. MUL methods typically rely on specialized forms of fine-tuning. Recent research has shown that data memorization is a key characteristic defining the difficulty of MUL. As a result, novel memorization-based unlearning methods have been developed, demonstrating exceptional performance with respect to unlearning quality, while maintaining high performance for model utility. Alas, these methods depend on knowing the memorization scores of data points and computing said scores is a notoriously time-consuming process. This in turn severely limits the scalability of these solutions and their practical impact for real-world applications. In this work, we tackle these scalability challenges of state-of-the-art memorization-based MUL algorithms using a series of memorization-score proxies. We first analyze the profiles of various proxies and then evaluate the performance of state-of-the-art (memorization-based) MUL algorithms in terms of both accuracy and privacy preservation. Our empirical results show that these proxies can introduce accuracy on par with full memorization-based unlearning while dramatically improving scalability. We view this work as an important step toward scalable and efficient machine unlearning.