# Ascent Fails to Forget arXiv: https://arxiv.org/abs/2509.26427 · HTML: https://mlsanity.com/paper/2509.26427/ ## Key facts - Cited by 12 papers in the map (data through 14 Oct 2026). - Placed in Machine unlearning, one of 12 directions in LLM safety and security. - Builds on 31 of its 42 references that are in the map. - First posted on arXiv 30 Sep 2025. ## Placed in - LLM safety and security / [Machine unlearning](https://mlsanity.com/safety/machine-unlearning/) ## Builds on (most cited first, 30 of 31) - [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015) - [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017) - [Machine Unlearning](https://mlsanity.com/paper/1912.03817/) (2019) - [The Implicit Bias of Gradient Descent on Separable Data](https://mlsanity.com/paper/1710.10345/) (2017) - [Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks](https://mlsanity.com/paper/1911.04933/) (2019) - [Certified Data Removal from Machine Learning Models](https://mlsanity.com/paper/1911.03030/) (2019) - [The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning](https://mlsanity.com/paper/2403.03218/) (2024) - [Making AI Forget You: Data Deletion in Machine Learning](https://mlsanity.com/paper/1907.05012/) (2019) - [Large Language Models Can Be Strong Differentially Private Learners](https://mlsanity.com/paper/2110.05679/) (2021) - [Who's Harry Potter? Approximate Unlearning in LLMs](https://mlsanity.com/paper/2310.02238/) (2023) - [Amnesiac Machine Learning](https://mlsanity.com/paper/2010.10981/) (2020) - [Ablating Concepts in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.13516/) (2023) - [Descent-to-Delete: Gradient-Based Methods for Machine Unlearning](https://mlsanity.com/paper/2007.02923/) (2020) - [MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions](https://mlsanity.com/paper/2305.14795/) (2023) - [Towards Unbounded Machine Unlearning](https://mlsanity.com/paper/2302.09880/) (2023) - [Approximate Data Deletion from Machine Learning Models](https://mlsanity.com/paper/2002.10077/) (2020) - [Machine Unlearning of Features and Labels](https://mlsanity.com/paper/2108.11577/) (2021) - [Model Sparsity Can Simplify Machine Unlearning](https://mlsanity.com/paper/2304.04934/) (2023) - [On the Necessity of Auditable Algorithmic Definitions for Machine Unlearning](https://mlsanity.com/paper/2110.11891/) (2021) - [BREEDS: Benchmarks for Subpopulation Shift](https://mlsanity.com/paper/2008.04859/) (2020) - [DeltaGrad: Rapid retraining of machine learning models](https://mlsanity.com/paper/2006.14755/) (2020) - [Linear Adversarial Concept Erasure](https://mlsanity.com/paper/2201.12091/) (2022) - [Towards Adversarial Evaluations for Inexact Machine Unlearning](https://mlsanity.com/paper/2201.06640/) (2022) - [Langevin Unlearning: A New Perspective of Noisy Gradient Descent for Machine Unlearning](https://mlsanity.com/paper/2401.10371/) (2024) - [To Each (Textual Sequence) Its Own: Improving Memorized-Data Unlearning in Large Language Models](https://mlsanity.com/paper/2405.03097/) (2024) - [Certifiable Machine Unlearning for Linear Models](https://mlsanity.com/paper/2106.15093/) (2021) - [Algorithms that Approximate Data Removal: New Results and Limitations](https://mlsanity.com/paper/2209.12269/) (2022) - [Machine Unlearning Fails to Remove Data Poisoning Attacks](https://mlsanity.com/paper/2406.17216/) (2024) - [Rewind-to-Delete: Certified Machine Unlearning for Nonconvex Functions](https://mlsanity.com/paper/2409.09778/) (2024) - [Attribute-to-Delete: Machine Unlearning via Datamodel Matching](https://mlsanity.com/paper/2410.23232/) (2024) ## Built on by (most cited first, 12 of 12) - [Locality-Aware Continual Unlearning for Diffusion Models](https://mlsanity.com/paper/2512.02657/) (2025) - [Class Unlearning via Depth-Aware Removal of Forget-Specific Directions](https://mlsanity.com/paper/2604.15166/) (2026) - [Forget by Uncertainty: Orthogonal Entropy Unlearning for Quantized Neural Networks](https://mlsanity.com/paper/2602.00567/) (2026) - [Beyond Forgetting: Representation Misdirection Elicits Controllable Side Behaviors and Capabilities](https://mlsanity.com/paper/2601.21702/) (2026) - [Temper-Then-Tilt: Principled Unlearning for Generative Models through Tempering and Classifier Guidance](https://mlsanity.com/paper/2602.10217/) (2026) - [Reference-Guided Machine Unlearning](https://mlsanity.com/paper/2603.11210/) (2026) - [Variance-Reduced (varepsilon,δ)-Unlearning using Forget Set Gradients](https://mlsanity.com/paper/2602.14938/) (2026) - [Behavioral Guarantees for Proxy-Based Unlearning](https://mlsanity.com/paper/2605.10680/) (2026) - [How Hard Can It Be? Hardness-Aware Multi-Objective Unlearning](https://mlsanity.com/paper/2606.02119/) (2026) - [TILDE: TILt-based Distributional Erasure for Concept Unlearning](https://mlsanity.com/paper/2607.06432/) (2026) - [LLM Unlearning for Cyber Defense: A Survey on Methods, Challenges, and Emerging Threats](https://mlsanity.com/paper/2607.16227/) (2026) - [When Is Deletion Ordering Tractable? From Update Dynamics to Permutation Structure](https://mlsanity.com/paper/2610.01149/) (2026) ## Abstract (arXiv) Contrary to common belief, we show that gradient ascent-based unconstrained optimization methods frequently fail to perform machine unlearning, a phenomenon we attribute to the inherent statistical dependence between the forget and retain data sets. This dependence, which can manifest itself even as simple correlations, undermines the misconception that these sets can be independently manipulated during unlearning. We provide empirical and theoretical evidence showing these methods often fail precisely due to this overlooked relationship. For random forget sets, this dependence means that degrading forget set metrics (which, for a retrained model, should mirror test set metrics) inevitably harms overall test performance. Going beyond random sets, we consider logistic regression as an instructive example where a critical failure mode emerges: inter-set dependence causes gradient descent-ascent iterations to progressively diverge from the ideal retrained model. Strikingly, these methods can converge to solutions that are not only far from the retrained ideal but are potentially even further from it than the original model itself, rendering the unlearning process actively detrimental. A toy example further illustrates how this dependence can trap models in inferior local minima, inescapable via finetuning. Our findings highlight that the presence of such statistical dependencies, even when manifest only as correlations, can be sufficient for ascent-based unlearning to fail. Our theoretical insights are corroborated by experiments on complex neural networks, demonstrating that these methods do not perform as expected in practice due to this unaddressed statistical interplay.