mlsanityData to 13 Oct 2026

Learning to Refuse: Towards Mitigating Privacy Risks in LLMs

arXiv 2407.10058 ↗First posted 14 Jul 2024cs.CL cs.AIcited by 22

On the map: LLM safety and security → Machine unlearning

Abstract (arXiv)

Large language models (LLMs) exhibit remarkable capabilities in understanding and generating natural language. However, these models can inadvertently memorize private information, posing significant privacy risks. This study addresses the challenge of enabling LLMs to protect specific individuals' private data without the need for complete retraining. We propose return, a Real-world pErsonal daTa UnleaRNing dataset, comprising 2,492 individuals from Wikipedia with associated QA pairs, to evaluate machine unlearning (MU) methods for protecting personal data in a realistic scenario. Additionally, we introduce the Name-Aware Unlearning Framework (NAUF) for Privacy Protection, which enables the model to learn which individuals' information should be protected without affecting its ability to answer questions related to other unrelated individuals. Our extensive experiments demonstrate that NAUF achieves a state-of-the-art average unlearning score, surpassing the best baseline method by 5.65 points, effectively protecting target individuals' personal data while maintaining the model's general capabilities.

Key facts. Cited by 22 papers in the map (data through 13 Oct 2026). Placed in Machine unlearning, one of 12 directions in LLM safety and security. Builds on 28 of its 47 references that are in the map. First posted on arXiv 14 Jul 2024.

Builds on

28 of 47 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning8 Apr
  2. 2024TOFU: A Task of Fictitious Unlearning for LLMs11 Jan
  3. 2023SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation19 Oct
  4. 2023Who's Harry Potter? Approximate Unlearning in LLMs3 Oct
  5. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Models4 Oct
  6. 2022Quark: Controllable Text Generation with Reinforced Unlearning26 May
  7. 2021Remember What You Want to Forget: Algorithms for Machine Unlearning4 Mar

Other references · 21

  1. 2020Language Models are Few-Shot Learnerscited by 32,320
  2. 2023GPT-4 Technical Reportcited by 18,278
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,986
  4. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,798
  5. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,843
  6. 2020Extracting Training Data from Large Language Modelscited by 2,181
  7. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,111
  8. 2019Machine Unlearningcited by 1,056
  9. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 973
  10. 2019Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networkscited by 617
  11. 2019Making AI Forget You: Data Deletion in Machine Learningcited by 462
  12. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 427
9 more
  1. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 314
  2. 2020Retrieve, Program, Repeat: Complex Knowledge Base Question Answering via Alternate Meta-learningcited by 216
  3. 2022Are Large Pre-Trained Language Models Leaking Your Personal Information?cited by 200
  4. 2023Can Sensitive Information Be Deleted From LLMs? Objectives for Defending Against Extraction Attackscited by 157
  5. 2021Federated Unlearning via Class-Discriminative Pruningcited by 115
  6. 2024Seal-Tools: Self-Instruct Tool Learning Dataset for Agent Tuning and Detailed Benchmarkcited by 63
  7. 2024Probing Language Models for Pre-training Data Detectioncited by 24
  8. 2024Dynamic Data Mixing Maximizes Instruction Tuning for Mixture-of-Expertscited by 19
  9. 2024Controllable and Diverse Data Augmentation with Large Language Model for Low-Resource Open-Domain Dialogue Generationcited by 4

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 3; 2025: 13; 2026: 6.

22 papers.

Sort:

By area:

  1. 2024A Closer Look at Machine Unlearning for Large Language Modelscited by 55
  2. 2025ReLearn: Unlearning via Learning for Large Language Modelscited by 21
  3. 2025Which Retain Set Matters for LLM Unlearning? A Case Study on Entity Unlearningcited by 14
  4. 2025Distillation Robustifies Unlearningcited by 11
  5. 2024Privacy-preserved LLM Cascade via CoT-enhanced Policy Learningcited by 8
  6. 2025R-TOFU: Unlearning in Large Reasoning Modelscited by 8
  7. 2024Regression for the Mean: Auto-Evaluation and Inference with Few Labels through Post-hoc Regressioncited by 6
  8. 2026Attention Smoothing Is All You Need For Unlearningcited by 6
  9. 2025UniErase: Towards Balanced and Precise Unlearning in Language Modelscited by 5
  10. 2025A Survey on Unlearning in Large Language Modelscited by 3
  11. 2025ZJUKLAB at SemEval-2025 Task 4: Unlearning via Model Mergingcited by 2
  12. 2025Measuring Physical-World Privacy Awareness of Large Language Models: An Evaluation Benchmarkcited by 2
10 more
  1. 2025Holistic Audit Dataset Generation for LLM Unlearning via Knowledge Graph Traversal and Redundancy Removalcited by 1
  2. 2025LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systemscited by 1
  3. 2025LLM Unlearning Under the Microscope: A Full-Stack View on Methods and Metricscited by 1
  4. 2026Lingua-SafetyBench: A Benchmark for Safety Evaluation of Multilingual Vision-Language Modelscited by 1
  5. 2026Unlearners Can Lie: Evaluating and Improving Honesty in LLM Unlearningcited by 1
  6. 2026Demystifying the Privacy-Utility Trade-off in LLM Interactionscited by 1
  7. 2025Malicious and Unintentional Disclosure Risks in Large Language Models for Code Generationcited by 0
  8. 2025Beyond Superficial Forgetting: Thorough Unlearning through Knowledge Density Estimation and Block Re-insertioncited by 0
  9. 2026CURaTE: Continual Unlearning in Real Time with Ensured Preservation of LLM Knowledgecited by 0
  10. 2026DualOptim+: Bridging Shared and Decoupled Optimizer States for Better Machine Unlearning in Large Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Preserving Privacy in Large Language Models: A Survey on Current Threats and Solutions
  2. 2026On the Recoverability of Private Information Unlearning in Large Language Models
  3. 2025A Survey on Privacy Risks and Protection in Large Language Models
  4. 2024Protecting Privacy Through Approximating Optimal Parameters for Sequence Unlearning in Language Models
  5. 2025Does Machine Unlearning Truly Remove Knowledge?
  6. 2026Data-Free Privacy-Preserving for LLMs via Model Inversion and Selective Unlearning
  7. 2025What Should LLMs Forget? Quantifying Personal Data in LLMs for Right-to-Be-Forgotten Requests
  8. 2024Machine Unlearning in Large Language Models