mlsanityData to 7 Oct 2026

Towards Natural Machine Unlearning

arXiv 2405.15495 ↗First posted 24 May 2024cs.LGcited by 11

On the map: LLM safety and security → Machine unlearning

Abstract (arXiv)

Machine unlearning (MU) aims to eliminate information that has been learned from specific training data, namely forgetting data, from a pre-trained model. Currently, the mainstream of existing MU methods involves modifying the forgetting data with incorrect labels and subsequently fine-tuning the model. While learning such incorrect information can indeed remove knowledge, the process is quite unnatural as the unlearning process undesirably reinforces the incorrect information and leads to over-forgetting. Towards more textitnatural machine unlearning, we inject correct information from the remaining data to the forgetting samples when changing their labels. Through pairing these adjusted samples with their labels, the model will tend to use the injected correct information and naturally suppress the information meant to be forgotten. Albeit straightforward, such a first step towards natural machine unlearning can significantly outperform current state-of-the-art approaches. In particular, our method substantially reduces the over-forgetting and leads to strong robustness to hyperparameters, making it a promising candidate for practical machine unlearning.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Machine unlearning, one of 12 directions in LLM safety and security. Builds on 53 of its 76 references that are in the map. First posted on arXiv 24 May 2024.

Builds on

53 of 76 references.

Closest ancestors · 19

Same direction, newest first

  1. 2024Challenging Forgets: Unveiling the Worst-Case Forget Sets in Machine Unlearning12 Mar
  2. 2024Remaining-data-free Machine Unlearning by Suppressing Sample Contribution23 Feb
  3. 2024Parameter-tuning-free data entry error unlearning with adaptive selective synaptic dampening6 Feb
  4. 2024Dataset Condensation Driven Machine Unlearning31 Jan
  5. 2023SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation19 Oct
  6. 2023Fast Machine Unlearning Without Retraining Through Selective Synaptic Dampening15 Aug
  7. 2023Machine Unlearning: Solutions and Challenges14 Aug
  8. 2023Model Sparsity Can Simplify Machine Unlearning11 Apr
  9. 2023Towards Unbounded Machine Unlearning20 Feb
  10. 2023Learning to Unlearn: Instance-wise Unlearning for Pre-trained Classifiers27 Jan
  11. 2022Can Bad Teaching Induce Forgetting? Unlearning in Deep Networks using an Incompetent Teacher17 May
  12. 2022Deep Unlearning via Randomized Conditionally Independent Hessians15 Apr
7 more
  1. 2022Towards Adversarial Evaluations for Inexact Machine Unlearning17 Jan
  2. 2022Zero-Shot Machine Unlearning14 Jan
  3. 2021Fast Yet Effective Machine Unlearning17 Nov
  4. 2021On the Necessity of Auditable Algorithmic Definitions for Machine Unlearning22 Oct
  5. 2021Unrolling SGD: Understanding Factors Influencing Machine Unlearning27 Sep
  6. 2021Machine Unlearning of Features and Labels26 Aug
  7. 2021Machine Unlearning via Algorithmic Stability25 Feb

Other references · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  5. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,690
  6. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  7. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  8. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  9. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
  10. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  11. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  12. 2015SegNet: A Deep Convolutional Encoder-Decoder Architecture for Image Segmentationcited by 1,483
22 more
  1. 2022Diffusion Models in Vision: A Surveycited by 1,008
  2. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  3. 2021Membership Inference Attacks From First Principlescited by 839
  4. 2019Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networkscited by 599
  5. 2019Certified Data Removal from Machine Learning Modelscited by 512
  6. 2020Graph Neural Networks in Recommender Systems: A Surveycited by 493
  7. 2019Making AI Forget You: Data Deletion in Machine Learningcited by 447
  8. 2014New insights and perspectives on the natural gradient methodcited by 435
  9. 2020Exploring Self-attention for Image Recognitioncited by 313
  10. 2020Amnesiac Machine Learningcited by 305
  11. 2020Systematic Evaluation of Privacy Risks of Machine Learning Modelscited by 273
  12. 2020Approximate Data Deletion from Machine Learning Modelscited by 250
  13. 2023SpectralGPT: Spectral Remote Sensing Foundation Modelcited by 250
  14. 2022TinyViT: Fast Pretraining Distillation for Small Vision Transformerscited by 232
  15. 2016Layer-wise Relevance Propagation for Neural Networks with Local Renormalization Layerscited by 197
  16. 2020Eigen-CAM: Class Activation Map using Principal Componentscited by 188
  17. 2021Efficient Training of Visual Transformers with Small Datasetscited by 107
  18. 2023Can Neural Network Memorization Be Localized?cited by 66
  19. 2022PUMA: Performance Unchanged Model Augmentation for Training Data Removalcited by 56
  20. 2021Software for Dataset-wide XAI: From Local Explanations to Global Insights with Zennit, CoRelAy, and ViRelAycited by 53
  21. 2022Repairing Neural Networks by Leaving the Right Past Behindcited by 27
  22. 2022A comparative study between vision transformers and CNNs in digital pathologycited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 3; 2026: 5.

11 papers.

  1. 2024Unlearning Personal Data from a Single Imagecited by 4
  2. 2025Geometric-disentangelment Unlearningcited by 4
  3. 2024MUSO: Achieving Exact Machine Unlearning in Over-Parameterized Regimescited by 3
  4. 2025Group-robust Machine Unlearningcited by 3
  5. 2026Class Unlearning via Depth-Aware Removal of Forget-Specific Directionscited by 3
  6. 2024Machine Unlearning using Forgetting Neural Networkscited by 2
  7. 2026Forget by Uncertainty: Orthogonal Entropy Unlearning for Quantized Neural Networkscited by 2
  8. 2026Representation Unlearning: Forgetting through Information Compressioncited by 1
  9. 2025Delete and Retain: Efficient Unlearning for Document Classificationcited by 0
  10. 2026SPACE: Source-free Proxy Anchor Concept Erasure for MLLMscited by 0
  11. 2026Signal-Guided Optimization for Machine Unlearningcited by 0