mlsanityData to 7 Oct 2026

UNDIAL: Self-Distillation with Adjusted Logits for Robust Unlearning in Large Language Models

arXiv 2402.10052 ↗First posted 15 Feb 2024cs.CL cs.AIcited by 49

On the map: LLM safety and security → Machine unlearning

Abstract (arXiv)

Mitigating the retention of sensitive or private information in large language models is essential for enhancing privacy and safety. Existing unlearning methods, like Gradient Ascent and Negative Preference Optimization, directly tune models to remove unwanted information. However, these methods often become unstable because they fine-tune by maximizing cross-entropy loss, which is the opposite of traditional loss minimization in learning. This reversal creates instability, especially on larger datasets, as the model struggles to balance unlearning with maintaining language capacity, leading to over-unlearning. In this paper, we introduce UnDIAL (Unlearning via Self-Distillation on Adjusted Logits), a novel and robust unlearning method. Our approach leverages self-distillation to adjust logits and selectively reduce the influence of targeted tokens. This technique ensures smooth convergence and avoids catastrophic forgetting, even in challenging unlearning tasks with large datasets and sequential unlearning requests. Extensive experiments show that UnDIAL can achieve both robustness in unlearning and scalability while maintaining stable training dynamics and resilience to hyperparameter tuning.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Placed in Machine unlearning, one of 12 directions in LLM safety and security. Builds on 37 of its 60 references that are in the map. First posted on arXiv 15 Feb 2024.

Builds on

37 of 60 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024TOFU: A Task of Fictitious Unlearning for LLMs11 Jan
  2. 2023Unlearn What You Want to Forget: Efficient Unlearning for LLMs31 Oct
  3. 2023Who's Harry Potter? Approximate Unlearning in LLMs3 Oct
  4. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Models4 Oct

Other references · 33

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  2. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  3. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  4. 2023Mistral 7Bcited by 3,065
  5. 2016Pointer Sentinel Mixture Modelscited by 2,992
  6. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,818
  7. 2016Deep Learning with Differential Privacycited by 2,682
  8. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  9. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  10. 2020Extracting Training Data from Large Language Modelscited by 2,144
  11. 2019Language Models as Knowledge Bases?cited by 1,922
  12. 2017Decoupled Weight Decay Regularizationcited by 1,722
21 more
  1. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  2. 2019Machine Unlearningcited by 1,020
  3. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  4. 2023Large Language Modelscited by 774
  5. 2021Deduplicating Training Data Makes Language Models Bettercited by 705
  6. 2023Bias and Fairness in Large Language Models: A Surveycited by 626
  7. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  8. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  9. 2019Certified Data Removal from Machine Learning Modelscited by 512
  10. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  11. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  12. 2019Making AI Forget You: Data Deletion in Machine Learningcited by 447
  13. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  14. 2021Large Language Models Can Be Strong Differentially Private Learnerscited by 405
  15. 2021Differentially Private Fine-tuning of Language Modelscited by 392
  16. 2022Deduplicating Training Data Mitigates Privacy Risks in Language Modelscited by 309
  17. 2020Differentially Private Learning Needs Better Features (or Much More Data)cited by 233
  18. 2022Quantifying Privacy Risks of Masked Language Models Using Membership Inference Attackscited by 168
  19. 2021How much do language models copy from their training data? Evaluating linguistic novelty in text generation using RAVENcited by 124
  20. 2020Variational Bayesian Unlearningcited by 114
  21. 2021Large Scale Private Learning via Low-rank Reparametrizationcited by 87

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 16; 2026: 30.

49 papers.

Sort:

By area:

  1. 2025CODI: Compressing Chain-of-Thought into Continuous Space via Self-Distillationcited by 216
  2. 2025OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metricscited by 80
  3. 2024LLM Unlearning via Loss Adjustment with Only Forget Datacited by 71
  4. 2024Machine Unlearning in Generative AI: A Surveycited by 49
  5. 2025Unlearning Isn't Deletion: Investigating Reversibility of Machine Unlearning in LLMscited by 41
  6. 2025Exploring Criteria of Loss Reweighting to Enhance LLM Unlearningcited by 39
  7. 2024Alternate Preference Optimization for Unlearning Factual Knowledge in Large Language Modelscited by 32
  8. 2025LLM Unlearning with LLM Beliefscited by 17
  9. 2025Distillation Robustifies Unlearningcited by 11
  10. 2025Constrained Entropic Unlearning: A Primal-Dual Framework for Large Language Modelscited by 10
  11. 2025Privacy-Aware Decoding: Mitigating Privacy Leakage of Large Language Models in Retrieval-Augmented Generationcited by 8
  12. 2025DRAGON: Guard LLM Unlearning in Context via Negative Detection and Reasoningcited by 7
37 more
  1. 2025Step-by-Step Reasoning Attack: Revealing 'Erased' Knowledge in Large Language Modelscited by 6
  2. 2026Attention Smoothing Is All You Need For Unlearningcited by 6
  3. 2026Weak-Driven Learning: How Weak Agents make Strong Agents Strongercited by 5
  4. 2025OFMU: Optimization-Driven Framework for Machine Unlearningcited by 4
  5. 2025Geometric-disentangelment Unlearningcited by 4
  6. 2026Contrastive Representation Shaping for LLM Unlearningcited by 4
  7. 2026REBEL: Hidden Knowledge Recovery via Evolutionary-Based Evaluation Loopcited by 4
  8. 2026Auditing Language Model Unlearning via Information Decompositioncited by 3
  9. 2026DUET: Distilled LLM Unlearning from an Efficiently Contextualized Teachercited by 3
  10. 2026Object Hallucination-Free Reinforcement Unlearning for Vision-Language Modelscited by 3
  11. 2026Visual-Noise Guided In-Context Distillation for Multimodal Large Language Model Unlearningcited by 3
  12. 2025Revisiting the Past: Data Unlearning with Model State Historycited by 2
  13. 2025Forget to Know, Remember to Use: Context-Aware Unlearning for Large Language Modelscited by 2
  14. 2026A Mechanistic Perspective and Circuit-Guided Difficulty Metric for Unlearningcited by 2
  15. 2026Randomized Antipodal Search Done Right for Data Pareto Improvement of LLM Unlearningcited by 2
  16. 2026Inference-Time Machine Unlearning via Gated Activation Redirectioncited by 2
  17. 2026Distinguishable Deletion: Unifying Knowledge Erasure and Refusal for Large Language Model Unlearningcited by 2
  18. 2026Divergence Decoding: Inference-Time Unlearning via Auxiliary Modelscited by 2
  19. 2025FALCON: Fine-grained Activation Manipulation by Contrastive Orthogonal Unalignment for Large Language Modelcited by 1
  20. 2025Feature-Selective Representation Misdirection for Machine Unlearningcited by 1
  21. 2026Measuring the Depth of LLM Unlearning via Activation Patchingcited by 1
  22. 2026On the Robustness of Machine Unlearning for Vision-Language Modelscited by 1
  23. 2026Learning What to Forget: Improving LLM Unlearning via Learned Token-Level Importancecited by 1
  24. 2026PreUnlearn: Auditing Collateral Knowledge Damage Before Large Language Model Unlearningcited by 1
  25. 2026SAU: Sparsity-Aware Unlearning for LLMs via Gradient Masking and Importance Redistributioncited by 0
  26. 2026EvoMU: Evolutionary Machine Unlearningcited by 0
  27. 2026MPU: Towards Secure and Privacy-Preserving Knowledge Unlearning for Large Language Modelscited by 0
  28. 2026Retain-Neutral Surrogates for Min-Max Unlearningcited by 0
  29. 2026SHRED: Retain-Set-Free Unlearning via Self-Distillation with Logit Demotioncited by 0
  30. 2026Fast Unlearning at Scale via Margin Self-Correctioncited by 0
  31. 2026AI Engram: In Search of Memory Traces in Artificial Intelligencecited by 0
  32. 2026RepSelect: Robust LLM Unlearning via Representation Selectivitycited by 0
  33. 2026SAGE: Retain-Aware Post-Hoc Sanitization of Final Unlearning Vectorcited by 0
  34. 2026What to Forget in Unlearning? Forget Set Curation for Language Modelscited by 0
  35. 2026ConceptGuard: Benchmarking Context-Sensitive Unlearning in Large Language Modelscited by 0
  36. 2026The Safeguard Worked. Is the LLM System Safer?cited by 0
  37. 2026UnlearningSoup: Is Repeated Tuning Necessary for Large Language Model Unlearning?cited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning
  2. 2024UnStar: Unlearning with Self-Taught Anti-Sample Reasoning for LLMs
  3. 2024Towards Robust and Parameter-Efficient Knowledge Unlearning for LLMs
  4. 2025Distribution Preference Optimization: A Fine-grained Perspective for LLM Unlearning
  5. 2026CALIBURN: Self-Calibrated LLM Unlearning Alignment
  6. 2023Large Language Model Unlearning
  7. 2025Invariance Makes LLM Unlearning Resilient Even to Unanticipated Downstream Fine-Tuning
  8. 2025Reveal and Release: Iterative LLM Unlearning with Self-generated Data