Distance Is Not Enough: Forget-Retain Alignment Gap Predicts LLM Relearning Robustness
On the map: LLM safety and security → Machine unlearning
Abstract (arXiv)
Machine unlearning aims to make a model forget specific data, yet unlearned LLMs often fail to stay unlearned: brief fine-tuning can revive removed knowledge. Existing robustness predictors rely on global weight-space displacement, but distance alone can be misleading when random or destructive updates collapse performance. We argue that relearning robustness depends on update structure: robust unlearning should affect forget-critical weights while sparing retain-critical ones. We introduce the Forget-Retain Alignment Gap (FRAG), a training-free predictor that scores an update's forget-retain alignment without running a relearning attack, and separates selective from dense updates more reliably than global distance. Building on the forget-critical, retain-sparing principle, Forget-Retain Pruning (FRP) improves relearning robustness. Our results suggest that weight selectivity better explains robustness than distance alone. Code is available at https://github.com/Yi1-Chen/FRAG.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Machine unlearning, one of 12 directions in LLM safety and security. Builds on 26 of its 33 references that are in the map. First posted on arXiv 26 Aug 2026.
Builds on
26 of 33 references.
Closest ancestors · 17
Same direction, newest first
- 2025OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metrics14 Jun
- 2025From Dormant to Deleted: Tamper-Resistant Unlearning Through Weight-Space Regularization28 May
- 2025Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyond7 Feb
- 2024Mechanistic Unlearning: Robust Knowledge Unlearning and Editing via Mechanistic Localization16 Oct
- 2024LLM Unlearning via Loss Adjustment with Only Forget Data14 Oct
- 2024Do Unlearning Methods Remove Information from Language Model Weights?11 Oct
- 2024Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning9 Oct
- 2024An Adversarial Perspective on Machine Unlearning for AI Safety26 Sep
- 2024MUSE: Machine Unlearning Six-Way Evaluation for Language Models8 Jul
- 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning8 Apr
- 2024Dissecting Language Models: Machine Unlearning via Selective Pruning2 Mar
- 2024TOFU: A Task of Fictitious Unlearning for LLMs11 Jan
5 more
- 2023SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation19 Oct
- 2023In-Context Unlearning: Language Models as Few Shot Unlearners11 Oct
- 2023Fast Machine Unlearning Without Retraining Through Selective Synaptic Dampening15 Aug
- 2023Model Sparsity Can Simplify Machine Unlearning11 Apr
- 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Models4 Oct
Other references · 9
- 2022Locating and Editing Factual Associations in GPTcited by 2,609
- 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
- 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
- 2023Zephyr: Direct Distillation of LM Alignmentcited by 498
- 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
- 2023Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attackscited by 156
- 2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 132
- 2024Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Spacecited by 88
- 2025Model Tampering Attacks Enable More Rigorous Evaluations of LLM Capabilitiescited by 36
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026CALIBURN: Self-Calibrated LLM Unlearning Alignment
- 2025Beyond Sharp Minima: Robust LLM Unlearning via Feedback-Guided Multi-Point Optimization
- 2026Why Some Models Resist Unlearning: A Linear Stability Perspective
- 2026RepSelect: Robust LLM Unlearning via Representation Selectivity
- 2026Faithful Dual-constrained Erasure for Robust LLM Safety Alignment
- 2025On the Impossibility of Retrain Equivalence in Machine Unlearning
- 2025LLM Unlearning Reveals a Stronger-Than-Expected Coreset Effect in Current Benchmarks
- 2026Aligned but Fragile: Enhancing LLM Safety Robustness via Zeroth-Order Optimization