GUARD: Guided Unlearning and Retention via Data Attribution for Large Language Models
On the map: LLM safety and security → Machine unlearning
Abstract (arXiv)
Unlearning large language models (LLMs) is important due to copyright and privacy concerns. A key challenge is unintended forgetting, where removing data inadvertently impairs model utility. While prior work has focused on architectural innovations, the influence of data-level factors remains underexplored. We introduce GUARD, a framework for **G**uided **U**nlearning **A**nd **R**etention via **D**ata Attribution. GUARD uses a lightweight proxy data attribution metric tailored to LLM unlearning, which efficiently quantifies the "alignment" between the Forget and Retain (or synthetic supplement) Sets. This metric leads to a new unlearning objective that assigns adaptive, nonuniform weights to samples, decaying exponentially with their proxy attribution scores, and thereby provably mitigates unintended retention loss. On the TOFU benchmark with a 10% forget set, applying GUARD to Gradient Ascent (GA) improves Retain Set ROUGE-L from 0.18 to 22.40 (a 124x increase) and Retain Probability from 0.06% to 17.50%, a substantial recovery from the GA collapse. Furthermore, GUARD combined with NPO (Negative Preference Optimization) outperforms Representation Misdirection for Unlearning (RMU) with 33.80% and 53.10% reductions in the Retain Set Sacrifice Rates computed from ROUGE-L and Probability (ρr and ρp), and a 16.01% improvement in knowledge retention on the MUSE NEWS Retain Set. These findings persist when attribution scores are computed using synthetic or highly subsampled (20%) Retain Sets, where NPO+GUARD still reduces ρr and ρp by up to 43.89% and 48.13% over NPO. Similar improvements are observed on MUSE, with a tolerable absolute privacy leakage degradation of 13.7 compared to the best-performing existing method in the privacy category.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Machine unlearning, one of 12 directions in LLM safety and security. Builds on 35 of its 53 references that are in the map. First posted on arXiv 12 Jun 2025.
Builds on
35 of 53 references.
Closest ancestors · 21
Same direction, newest first
- 2025Do LLMs Really Forget? Evaluating Unlearning with Knowledge Correlation and Confidence Awareness6 Jun
- 2025Exploring Criteria of Loss Reweighting to Enhance LLM Unlearning17 May
- 2025Rethinking LLM Unlearning Objectives: A Gradient Perspective and Go Beyond26 Feb
- 2024WAGLE: Strategic Weight Attribution for Effective and Modular Unlearning in Large Language Models23 Oct
- 2024Catastrophic Failure of LLM Unlearning via Quantization21 Oct
- 2024LLM Unlearning via Loss Adjustment with Only Forget Data14 Oct
- 2024Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning9 Oct
- 2024MUSE: Machine Unlearning Six-Way Evaluation for Language Models8 Jul
- 2024Towards Effective Evaluations and Comparisons for LLM Unlearning Methods13 Jun
- 2024Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Difference12 Jun
- 2024Large Language Model Unlearning via Embedding-Corrupted Prompts12 Jun
- 2024Single Image Unlearning: Efficient Machine Unlearning in Multimodal Large Language Models21 May
9 more
- 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning8 Apr
- 2024Certified Machine Unlearning via Noisy Stochastic Gradient Descent25 Mar
- 2024Machine Unlearning of Pre-trained Large Language Models23 Feb
- 2024TOFU: A Task of Fictitious Unlearning for LLMs11 Jan
- 2023Large Language Model Unlearning14 Oct
- 2022Quark: Controllable Text Generation with Reinforced Unlearning26 May
- 2022Continual Learning and Private Unlearning24 Mar
- 2022Zero-Shot Machine Unlearning14 Jan
- 2021Unrolling SGD: Understanding Factors Influencing Machine Unlearning27 Sep
Other references · 14
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
- 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
- 2020Extracting Training Data from Large Language Modelscited by 2,189
- 2017Understanding Black-box Predictions via Influence Functionscited by 1,741
- 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
- 2020Influence Functions in Deep Learning Are Fragilecited by 252
- 2022If Influence Functions are the Answer, Then What is the Question?cited by 144
- 2019Sanity Checks for Saliency Metricscited by 119
- 2024Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Spacecited by 91
- 2024Mini-batch Coresets for Memory-efficient Language Model Training on Data Mixturescited by 12
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2025LLM Unlearning on Noisy Forget Sets: A Study of Incomplete, Rewritten, and Watermarked Data
- 2025Not Every Token Needs Forgetting: Selective Unlearning to Limit Change in Utility in Large Language Model Unlearning
- 2025DRAGON: Guard LLM Unlearning in Context via Negative Detection and Reasoning
- 2026ConceptGuard: Benchmarking Context-Sensitive Unlearning in Large Language Models
- 2025GUARD: Generation-time LLM Unlearning via Adaptive Restriction and Detection
- 2023Unlearn What You Want to Forget: Efficient Unlearning for LLMs
- 2025OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metrics
- 2026Forgetting Only What Matters: Layer-Selective Unlearning toward Robust LLMs