# Learning to Refuse: Towards Mitigating Privacy Risks in LLMs

arXiv: https://arxiv.org/abs/2407.10058 · HTML: https://mlsanity.com/paper/2407.10058/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in Machine unlearning, one of 12 directions in LLM safety and security.
- Builds on 28 of its 47 references that are in the map.
- First posted on arXiv 14 Jul 2024.

## Placed in

- LLM safety and security / [Machine unlearning](https://mlsanity.com/safety/machine-unlearning/)

## Builds on (most cited first, 28 of 28)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge](https://mlsanity.com/paper/1803.05457/) (2018)
- [PIQA: Reasoning about Physical Commonsense in Natural Language](https://mlsanity.com/paper/1911.11641/) (2019)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories](https://mlsanity.com/paper/2212.10511/) (2022)
- [Machine Unlearning](https://mlsanity.com/paper/1912.03817/) (2019)
- [The LAMBADA dataset: Word prediction requiring a broad discourse context](https://mlsanity.com/paper/1606.06031/) (2016)
- [Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks](https://mlsanity.com/paper/1911.04933/) (2019)
- [TOFU: A Task of Fictitious Unlearning for LLMs](https://mlsanity.com/paper/2401.06121/) (2024)
- [Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning](https://mlsanity.com/paper/2404.05868/) (2024)
- [Making AI Forget You: Data Deletion in Machine Learning](https://mlsanity.com/paper/1907.05012/) (2019)
- [Knowledge Unlearning for Mitigating Privacy Risks in Language Models](https://mlsanity.com/paper/2210.01504/) (2022)
- [Scalable Extraction of Training Data from (Production) Language Models](https://mlsanity.com/paper/2311.17035/) (2023)
- [SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation](https://mlsanity.com/paper/2310.12508/) (2023)
- [Who's Harry Potter? Approximate Unlearning in LLMs](https://mlsanity.com/paper/2310.02238/) (2023)
- [Remember What You Want to Forget: Algorithms for Machine Unlearning](https://mlsanity.com/paper/2103.03279/) (2021)
- [Deduplicating Training Data Mitigates Privacy Risks in Language Models](https://mlsanity.com/paper/2202.06539/) (2022)
- [Quark: Controllable Text Generation with Reinforced Unlearning](https://mlsanity.com/paper/2205.13636/) (2022)
- [Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learning](https://mlsanity.com/paper/2010.15875/) (2020)
- [Are Large Pre-Trained Language Models Leaking Your Personal Information?](https://mlsanity.com/paper/2205.12628/) (2022)
- [Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attacks](https://mlsanity.com/paper/2309.17410/) (2023)
- [Federated Unlearning via Class-Discriminative Pruning](https://mlsanity.com/paper/2110.11794/) (2021)
- [Seal-Tools: Self-Instruct Tool Learning Dataset for Agent Tuning and Detailed Benchmark](https://mlsanity.com/paper/2405.08355/) (2024)
- [Probing Language Models for Pre-training Data Detection](https://mlsanity.com/paper/2406.01333/) (2024)
- [Dynamic Data Mixing Maximizes Instruction Tuning for Mixture-of-Experts](https://mlsanity.com/paper/2406.11256/) (2024)
- [Controllable and Diverse Data Augmentation with Large Language Model for Low-Resource Open-Domain Dialogue Generation](https://mlsanity.com/paper/2404.00361/) (2024)

## Built on by (most cited first, 23 of 23)

- [A Closer Look at Machine Unlearning for Large Language Models](https://mlsanity.com/paper/2410.08109/) (2024)
- [ReLearn: Unlearning via Learning for Large Language Models](https://mlsanity.com/paper/2502.11190/) (2025)
- [Which Retain Set Matters for LLM Unlearning? A Case Study on Entity Unlearning](https://mlsanity.com/paper/2502.11441/) (2025)
- [Distillation Robustifies Unlearning](https://mlsanity.com/paper/2506.06278/) (2025)
- [R-TOFU: Unlearning in Large Reasoning Models](https://mlsanity.com/paper/2505.15214/) (2025)
- [Privacy-preserved LLM Cascade via CoT-enhanced Policy Learning](https://mlsanity.com/paper/2410.08014/) (2024)
- [Regression for the Mean: Auto-Evaluation and Inference with Few Labels through Post-hoc Regression](https://mlsanity.com/paper/2411.12665/) (2024)
- [Attention Smoothing Is All You Need For Unlearning](https://mlsanity.com/paper/2603.01285/) (2026)
- [UniErase: Towards Balanced and Precise Unlearning in Language Models](https://mlsanity.com/paper/2505.15674/) (2025)
- [A Survey on Unlearning in Large Language Models](https://mlsanity.com/paper/2510.25117/) (2025)
- [ZJUKLAB at SemEval-2025 Task 4: Unlearning via Model Merging](https://mlsanity.com/paper/2503.21088/) (2025)
- [Measuring Physical-World Privacy Awareness of Large Language Models: An Evaluation Benchmark](https://mlsanity.com/paper/2510.02356/) (2025)
- [LLM Unlearning Under the Microscope: A Full-Stack View on Methods and Metrics](https://mlsanity.com/paper/2510.07626/) (2025)
- [Lingua-SafetyBench: A Benchmark for Safety Evaluation of Multilingual Vision-Language Models](https://mlsanity.com/paper/2601.22737/) (2026)
- [Holistic Audit Dataset Generation for LLM Unlearning via Knowledge Graph Traversal and Redundancy Removal](https://mlsanity.com/paper/2502.18810/) (2025)
- [LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systems](https://mlsanity.com/paper/2509.10682/) (2025)
- [Unlearners Can Lie: Evaluating and Improving Honesty in LLM Unlearning](https://mlsanity.com/paper/2605.08765/) (2026)
- [Demystifying the Privacy-Utility Trade-off in LLM Interactions](https://mlsanity.com/paper/2609.10992/) (2026)
- [Malicious and Unintentional Disclosure Risks in Large Language Models for Code Generation](https://mlsanity.com/paper/2503.22760/) (2025)
- [Beyond Superficial Forgetting: Thorough Unlearning through Knowledge Density Estimation and Block Re-insertion](https://mlsanity.com/paper/2511.11667/) (2025)
- [CURaTE: Continual Unlearning in Real Time with Ensured Preservation of LLM Knowledge](https://mlsanity.com/paper/2604.14644/) (2026)
- [DualOptim+: Bridging Shared and Decoupled Optimizer States for Better Machine Unlearning in Large Language Models](https://mlsanity.com/paper/2605.21539/) (2026)
- [REDaCT: Collapse-Free Unlearning in Large Reasoning Models](https://mlsanity.com/paper/2610.12769/) (2026)

## Abstract (arXiv)

Large language models (LLMs) exhibit remarkable capabilities in understanding and generating natural language. However, these models can inadvertently memorize private information, posing significant privacy risks. This study addresses the challenge of enabling LLMs to protect specific individuals' private data without the need for complete retraining. We propose return, a Real-world pErsonal daTa UnleaRNing dataset, comprising 2,492 individuals from Wikipedia with associated QA pairs, to evaluate machine unlearning (MU) methods for protecting personal data in a realistic scenario. Additionally, we introduce the Name-Aware Unlearning Framework (NAUF) for Privacy Protection, which enables the model to learn which individuals' information should be protected without affecting its ability to answer questions related to other unrelated individuals. Our extensive experiments demonstrate that NAUF achieves a state-of-the-art average unlearning score, surpassing the best baseline method by 5.65 points, effectively protecting target individuals' personal data while maintaining the model's general capabilities.