mlsanityData to 7 Oct 2026

Label Smoothing Improves Machine Unlearning

arXiv 2406.07698 ↗First posted 11 Jun 2024cs.LGcited by 8

On the map: LLM safety and security → Machine unlearning

Abstract (arXiv)

The objective of machine unlearning (MU) is to eliminate previously learned data from a model. However, it is challenging to strike a balance between computation cost and performance when using existing MU techniques. Taking inspiration from the influence of label smoothing on model confidence and differential privacy, we propose a simple gradient-based MU approach that uses an inverse process of label smoothing. This work introduces UGradSL, a simple, plug-and-play MU approach that uses smoothed labels. We provide theoretical analyses demonstrating why properly introducing label smoothing improves MU performance. We conducted extensive experiments on six datasets of various sizes and different modalities, demonstrating the effectiveness and robustness of our proposed method. The consistent improvement in MU performance is only at a marginal cost of additional computations. For instance, UGradSL improves over the gradient ascent MU baseline by 66% unlearning accuracy without sacrificing unlearning efficiency.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in Machine unlearning, one of 12 directions in LLM safety and security. Builds on 60 of its 81 references that are in the map. First posted on arXiv 11 Jun 2024.

Builds on

60 of 81 references.

Closest ancestors · 11

Same direction, newest first

  1. 2023Unlearning via Sparse Representations26 Nov
  2. 2023SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation19 Oct
  3. 2023Towards Unbounded Machine Unlearning20 Feb
  4. 2023Learning to Unlearn: Instance-wise Unlearning for Pre-trained Classifiers27 Jan
  5. 2022A Survey of Machine Unlearning6 Sep
  6. 2022Evaluating Machine Unlearning via Epistemic Uncertainty23 Aug
  7. 2021Unrolling SGD: Understanding Factors Influencing Machine Unlearning27 Sep
  8. 2021Hard to Forget: Poisoning Attacks on Certified Machine Unlearning17 Sep
  9. 2021Machine Unlearning of Features and Labels26 Aug
  10. 2021Remember What You Want to Forget: Algorithms for Machine Unlearning4 Mar
  11. 2021Machine Unlearning via Algorithmic Stability25 Feb

Other references · 49

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  6. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  7. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  8. 2016Deep Learning with Differential Privacycited by 2,682
  9. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  10. 2020Extracting Training Data from Large Language Modelscited by 2,144
  11. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  12. 2019Machine Unlearningcited by 1,020
37 more
  1. 2019When Does Label Smoothing Help?cited by 985
  2. 2021Membership Inference Attacks From First Principlescited by 839
  3. 2019Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networkscited by 599
  4. 2019Certified Data Removal from Machine Learning Modelscited by 512
  5. 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 461
  6. 2019Making AI Forget You: Data Deletion in Machine Learningcited by 447
  7. 2020Label-Only Membership Inference Attackscited by 371
  8. 2020Amnesiac Machine Learningcited by 305
  9. 2020Descent-to-Delete: Gradient-Based Methods for Machine Unlearningcited by 303
  10. 2020Systematic Evaluation of Privacy Risks of Machine Learning Modelscited by 273
  11. 2019White-box vs Black-box: Bayes Optimal Strategies for Membership Inferencecited by 265
  12. 2020Approximate Data Deletion from Machine Learning Modelscited by 250
  13. 2021Enhanced Membership Inference Attacks against Machine Learning Modelscited by 237
  14. 2020Does label smoothing mitigate label noise?cited by 204
  15. 2020When Machine Unlearning Jeopardizes Privacycited by 181
  16. 2019Stolen Memories: Leveraging Model Memorization for Calibrated White-Box Membership Inferencecited by 168
  17. 2020Membership Leakage in Label-Only Exposurescited by 156
  18. 2019On the Accuracy of Influence Functions for Measuring Group Effectscited by 146
  19. 2021Graph Unlearningcited by 129
  20. 2023Privacy Auditing with One (1) Training Runcited by 119
  21. 2022The Right to be Forgotten in Federated Learning: An Efficient Realization with Rapid Retrainingcited by 109
  22. 2021Federated Unlearning via Class-Discriminative Pruningcited by 108
  23. 2016Towards better decoding and language model integration in sequence to sequence modelscited by 100
  24. 2023Low-Cost High-Power Membership Inference Attackscited by 99
  25. 2018Understanding the Origins of Bias in Word Embeddingscited by 90
  26. 2019On Second-Order Group Influence Functions for Black-Box Predictionscited by 78
  27. 2021To Smooth or Not? When Label Smoothing Meets Noisy Labelscited by 70
  28. 2019Membership Inference Attacks on Sequence-to-Sequence Models: Is My Data In Your Machine Translation System?cited by 68
  29. 2014Differential Privacy and Machine Learning: a Survey and Reviewcited by 66
  30. 2022Backdoor Defense with Machine Unlearningcited by 61
  31. 2022Certified Graph Unlearningcited by 46
  32. 2022Hidden Poison: Machine Unlearning Enables Camouflaged Poisoning Attackscited by 37
  33. 2022A Data-Based Perspective on Transfer Learningcited by 32
  34. 2022Understanding Instance-Level Impact of Fairness Constraintscited by 27
  35. 2022Fair Infinitesimal Jackknife: Mitigating the Influence of Biased Training Data Points Without Refittingcited by 26
  36. 2020Selective Forgetting of Deep Networks at a Finer Level than Samplescited by 12
  37. 2021Out-of-Distribution Generalization Analysis via Influence Functioncited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 5; 2026: 2.

8 papers.

  1. 2024LLM Unlearning via Loss Adjustment with Only Forget Datacited by 71
  2. 2025Exploring Criteria of Loss Reweighting to Enhance LLM Unlearningcited by 39
  3. 2025GUARD: Generation-time LLM Unlearning via Adaptive Restriction and Detectioncited by 17
  4. 2025RULE: Reinforcement UnLEarning Achieves Forget-Retain Pareto Optimalitycited by 14
  5. 2026Explainable LLM Unlearning Through Reasoningcited by 7
  6. 2025Analise de Desaprendizado de Maquina em Modelos de Classificacao de Imagens Medicascited by 1
  7. 2025Data Augmentation Improves Machine Unlearningcited by 0
  8. 2026Anti-Backdoor Coreset Selection via Cumulative Entropycited by 0