# RKLD: Reverse KL-Divergence-based Knowledge Distillation for Unlearning Personal Information in Large Language Models

arXiv: https://arxiv.org/abs/2406.01983 · HTML: https://mlsanity.com/paper/2406.01983/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Placed in Machine unlearning, one of 12 directions in LLM safety and security.
- Builds on 23 of its 36 references that are in the map.
- First posted on arXiv 4 Jun 2024.

## Placed in

- LLM safety and security / [Machine unlearning](https://mlsanity.com/safety/machine-unlearning/)
- Knowledge distillation / [Knowledge distillation for LLMs](https://mlsanity.com/distill/knowledge-distillation-for-llms/)

## Builds on (most cited first, 23 of 23)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Machine Unlearning](https://mlsanity.com/paper/1912.03817/) (2019)
- [Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks](https://mlsanity.com/paper/1911.04933/) (2019)
- [TOFU: A Task of Fictitious Unlearning for LLMs](https://mlsanity.com/paper/2401.06121/) (2024)
- [Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning](https://mlsanity.com/paper/2404.05868/) (2024)
- [Knowledge Unlearning for Mitigating Privacy Risks in Language Models](https://mlsanity.com/paper/2210.01504/) (2022)
- [Who's Harry Potter? Approximate Unlearning in LLMs](https://mlsanity.com/paper/2310.02238/) (2023)
- [Remember What You Want to Forget: Algorithms for Machine Unlearning](https://mlsanity.com/paper/2103.03279/) (2021)
- [Forget-Me-Not: Learning to Forget in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2303.17591/) (2023)
- [Quark: Controllable Text Generation with Reinforced Unlearning](https://mlsanity.com/paper/2205.13636/) (2022)
- [In-Context Unlearning: Language Models as Few Shot Unlearners](https://mlsanity.com/paper/2310.07579/) (2023)
- [Unlearn What You Want to Forget: Efficient Unlearning for LLMs](https://mlsanity.com/paper/2310.20150/) (2023)
- [Dataset Inference: Ownership Resolution in Machine Learning](https://mlsanity.com/paper/2104.10706/) (2021)
- [KGA: A General Machine Unlearning Framework Based on Knowledge Gap Alignment](https://mlsanity.com/paper/2305.06535/) (2023)
- [Federated Unlearning: How to Efficiently Erase a Client in FL?](https://mlsanity.com/paper/2207.05521/) (2022)
- [Privacy Issues in Large Language Models: A Survey](https://mlsanity.com/paper/2312.06717/) (2023)
- [uChecker: Masked Pretrained Language Models as Unsupervised Chinese Spelling Checkers](https://mlsanity.com/paper/2209.07068/) (2022)
- [PhysNLU: A Language Resource for Evaluating Natural Language Understanding and Explanation Coherence in Physics](https://mlsanity.com/paper/2201.04275/) (2022)
- [Gradient Ascent Post-training Enhances Language Model Generalization](https://mlsanity.com/paper/2306.07052/) (2023)
- [f-Divergence Inequalities via Functional Domination](https://mlsanity.com/paper/1610.09110/) (2016)

## Built on by (most cited first, 21 of 21)

- [Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning](https://mlsanity.com/paper/2410.07163/) (2024)
- [A Closer Look at Machine Unlearning for Large Language Models](https://mlsanity.com/paper/2410.08109/) (2024)
- [LLM Collaboration With Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2508.04652/) (2025)
- [Meta-Unlearning on Diffusion Models: Preventing Relearning Unlearned Concepts](https://mlsanity.com/paper/2410.12777/) (2024)
- [A Comprehensive Survey of Machine Unlearning Techniques for Large Language Models](https://mlsanity.com/paper/2503.01854/) (2025)
- [Reasoning Model Unlearning: Forgetting Traces, Not Just Answers, While Preserving Reasoning Skills](https://mlsanity.com/paper/2506.12963/) (2025)
- [SAUCE: Selective Concept Unlearning in Vision-Language Models with Sparse Autoencoders](https://mlsanity.com/paper/2503.14530/) (2025)
- [Distillation Robustifies Unlearning](https://mlsanity.com/paper/2506.06278/) (2025)
- [Does Machine Unlearning Truly Remove Knowledge?](https://mlsanity.com/paper/2505.23270/) (2025)
- [Attention Smoothing Is All You Need For Unlearning](https://mlsanity.com/paper/2603.01285/) (2026)
- [Step-by-Step Reasoning Attack: Revealing 'Erased' Knowledge in Large Language Models](https://mlsanity.com/paper/2506.17279/) (2025)
- [Erasing Without Remembering: Implicit Knowledge Forgetting in Large Language Models](https://mlsanity.com/paper/2502.19982/) (2025)
- [REBEL: Hidden Knowledge Recovery via Evolutionary-Based Evaluation Loop](https://mlsanity.com/paper/2602.06248/) (2026)
- [A More Practical Approach to Machine Unlearning](https://mlsanity.com/paper/2406.09391/) (2024)
- [Efficient Verified Machine Unlearning For Distillation](https://mlsanity.com/paper/2503.22539/) (2025)
- [AUVIC: Adversarial Unlearning of Visual Concepts for Multi-modal Large Language Models](https://mlsanity.com/paper/2511.11299/) (2025)
- [Lifelong Learning with Behavior Consolidation for Vehicle Routing](https://mlsanity.com/paper/2509.21765/) (2025)
- [Erase to Retain: Low Rank Adaptation Guided Selective Unlearning in Medical Segmentation Networks](https://mlsanity.com/paper/2511.16574/) (2025)
- [EvoMU: Evolutionary Machine Unlearning](https://mlsanity.com/paper/2602.02139/) (2026)
- [SHRED: Retain-Set-Free Unlearning via Self-Distillation with Logit Demotion](https://mlsanity.com/paper/2605.07482/) (2026)
- [Fast Unlearning at Scale via Margin Self-Correction](https://mlsanity.com/paper/2606.02920/) (2026)

## Abstract (arXiv)

With the passage of the Right to Be Forgotten (RTBF) regulations and the scaling up of language model training datasets, research on model unlearning in large language models (LLMs) has become more crucial. Before the era of LLMs, machine unlearning research focused mainly on classification tasks in models with small parameters. In these tasks, the content to be forgotten or retained is clear and straightforward. However, as parameter sizes have grown and tasks have become more complex, balancing forget quality and model utility has become more challenging, especially in scenarios involving personal data instead of classification results. Existing methods based on gradient ascent and its variants often struggle with this balance, leading to unintended information loss or partial forgetting. To address this challenge, we propose RKLD, a novel Reverse KL-Divergence-based Knowledge Distillation unlearning algorithm for LLMs targeting the unlearning of personal information. Through RKLD, we achieve significant forget quality and effectively maintain the model utility in our experiments.