# GUARD: Guided Unlearning and Retention via Data Attribution for Large Language Models

arXiv: https://arxiv.org/abs/2506.10946 · HTML: https://mlsanity.com/paper/2506.10946/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Machine unlearning, one of 12 directions in LLM safety and security.
- Builds on 35 of its 53 references that are in the map.
- First posted on arXiv 12 Jun 2025.

## Placed in

- LLM safety and security / [Machine unlearning](https://mlsanity.com/safety/machine-unlearning/)

## Builds on (most cited first, 30 of 35)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Finetuned Language Models Are Zero-Shot Learners](https://mlsanity.com/paper/2109.01652/) (2021)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [Textbooks Are All You Need II: phi-1.5 technical report](https://mlsanity.com/paper/2309.05463/) (2023)
- [TOFU: A Task of Fictitious Unlearning for LLMs](https://mlsanity.com/paper/2401.06121/) (2024)
- [Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning](https://mlsanity.com/paper/2404.05868/) (2024)
- [Large Language Model Unlearning](https://mlsanity.com/paper/2310.10683/) (2023)
- [Unrolling SGD: Understanding Factors Influencing Machine Unlearning](https://mlsanity.com/paper/2109.13398/) (2021)
- [MUSE: Machine Unlearning Six-Way Evaluation for Language Models](https://mlsanity.com/paper/2407.06460/) (2024)
- [Influence Functions in Deep Learning Are Fragile](https://mlsanity.com/paper/2006.14651/) (2020)
- [Quark: Controllable Text Generation with Reinforced Unlearning](https://mlsanity.com/paper/2205.13636/) (2022)
- [Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning](https://mlsanity.com/paper/2410.07163/) (2024)
- [If Influence Functions are the Answer, Then What is the Question?](https://mlsanity.com/paper/2209.05364/) (2022)
- [Zero-Shot Machine Unlearning](https://mlsanity.com/paper/2201.05629/) (2022)
- [Continual Learning and Private Unlearning](https://mlsanity.com/paper/2203.12817/) (2022)
- [Sanity Checks for Saliency Metrics](https://mlsanity.com/paper/1912.01451/) (2019)
- [Machine Unlearning of Pre-trained Large Language Models](https://mlsanity.com/paper/2402.15159/) (2024)
- [Large Language Model Unlearning via Embedding-Corrupted Prompts](https://mlsanity.com/paper/2406.07933/) (2024)
- [Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Space](https://mlsanity.com/paper/2402.09063/) (2024)
- [Reversing the Forget-Retain Objectives: An Efficient LLM Unlearning Framework from Logit Difference](https://mlsanity.com/paper/2406.08607/) (2024)
- [Catastrophic Failure of LLM Unlearning via Quantization](https://mlsanity.com/paper/2410.16454/) (2024)
- [LLM Unlearning via Loss Adjustment with Only Forget Data](https://mlsanity.com/paper/2410.11143/) (2024)
- [Rethinking LLM Unlearning Objectives: A Gradient Perspective and Go Beyond](https://mlsanity.com/paper/2502.19301/) (2025)
- [Single Image Unlearning: Efficient Machine Unlearning in Multimodal Large Language Models](https://mlsanity.com/paper/2405.12523/) (2024)
- [Exploring Criteria of Loss Reweighting to Enhance LLM Unlearning](https://mlsanity.com/paper/2505.11953/) (2025)
- [Towards Effective Evaluations and Comparisons for LLM Unlearning Methods](https://mlsanity.com/paper/2406.09179/) (2024)
- [WAGLE: Strategic Weight Attribution for Effective and Modular Unlearning in Large Language Models](https://mlsanity.com/paper/2410.17509/) (2024)

## Built on by (most cited first, 2 of 2)

- [Making LLMs Truly Forget: Deep Unlearning by Searching, Selecting, and Severing Knowledge Paths](https://mlsanity.com/paper/2609.34442/) (2026)
- [Decouple to Forget: Semantic Mediation for Sensitive Unlearning in Large Language Models](https://mlsanity.com/paper/2610.17654/) (2026)

## Abstract (arXiv)

Unlearning large language models (LLMs) is important due to copyright and privacy concerns. A key challenge is unintended forgetting, where removing data inadvertently impairs model utility. While prior work has focused on architectural innovations, the influence of data-level factors remains underexplored. We introduce GUARD, a framework for **G**uided **U**nlearning **A**nd **R**etention via **D**ata Attribution. GUARD uses a lightweight proxy data attribution metric tailored to LLM unlearning, which efficiently quantifies the "alignment" between the Forget and Retain (or synthetic supplement) Sets. This metric leads to a new unlearning objective that assigns adaptive, nonuniform weights to samples, decaying exponentially with their proxy attribution scores, and thereby provably mitigates unintended retention loss. On the TOFU benchmark with a 10% forget set, applying GUARD to Gradient Ascent (GA) improves Retain Set ROUGE-L from 0.18 to 22.40 (a 124x increase) and Retain Probability from 0.06% to 17.50%, a substantial recovery from the GA collapse. Furthermore, GUARD combined with NPO (Negative Preference Optimization) outperforms Representation Misdirection for Unlearning (RMU) with 33.80% and 53.10% reductions in the Retain Set Sacrifice Rates computed from ROUGE-L and Probability (ρr and ρp), and a 16.01% improvement in knowledge retention on the MUSE NEWS Retain Set. These findings persist when attribution scores are computed using synthetic or highly subsampled (20%) Retain Sets, where NPO+GUARD still reduces ρr and ρp by up to 43.89% and 48.13% over NPO. Similar improvements are observed on MUSE, with a tolerable absolute privacy leakage degradation of 13.7 compared to the best-performing existing method in the privacy category.