mlsanityData to 7 Oct 2026

Machine Unlearning of Features and Labels

arXiv 2108.11577 ↗First posted 26 Aug 2021cs.LG cs.CRcited by 223

On the map: LLM safety and security → Machine unlearning

Abstract (arXiv)

Removing information from a machine learning model is a non-trivial task that requires to partially revert the training process. This task is unavoidable when sensitive data, such as credit card numbers or passwords, accidentally enter the model and need to be removed afterwards. Recently, different concepts for machine unlearning have been proposed to address this problem. While these approaches are effective in removing individual data points, they do not scale to scenarios where larger groups of features and labels need to be reverted. In this paper, we propose the first method for unlearning features and labels. Our approach builds on the concept of influence functions and realizes unlearning through closed-form updates of model parameters. It enables to adapt the influence of training data on a learning model retrospectively, thereby correcting data leaks and privacy issues. For learning models with strongly convex loss functions, our method provides certified unlearning with theoretical guarantees. For models with non-convex losses, we empirically show that unlearning features and labels is effective and significantly faster than other strategies.

Key facts. Cited by 223 papers in the map (data through 7 Oct 2026). Placed in Machine unlearning, one of 12 directions in LLM safety and security. Builds on 32 of its 81 references that are in the map. First posted on arXiv 26 Aug 2021.

Builds on

32 of 81 references.

References · 32

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2016Deep Learning with Differential Privacycited by 2,682
  4. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  5. 2020Extracting Training Data from Large Language Modelscited by 2,144
  6. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  7. 2019Machine Unlearningcited by 1,020
  8. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  9. 2019Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networkscited by 599
  10. 2013Generating Sequences With Recurrent Neural Networkscited by 549
  11. 2019Certified Data Removal from Machine Learning Modelscited by 512
  12. 2018ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Modelscited by 475
20 more
  1. 2019Making AI Forget You: Data Deletion in Machine Learningcited by 447
  2. 2018Lipschitz regularity of deep neural networks: analysis and efficient estimationcited by 413
  3. 2018Regularisation of Neural Networks by Enforcing Lipschitz Continuitycited by 311
  4. 2020Descent-to-Delete: Gradient-Based Methods for Machine Unlearningcited by 303
  5. 2019Efficient and Accurate Estimation of Lipschitz Constants for Deep Neural Networkscited by 290
  6. 2020Influence Functions in Deep Learning Are Fragilecited by 247
  7. 2019Stolen Memories: Leveraging Model Memorization for Calibrated White-Box Membership Inferencecited by 168
  8. 2020Mixed-Privacy Forgetting in Deep Networkscited by 158
  9. 2019On the Accuracy of Influence Functions for Measuring Group Effectscited by 146
  10. 2020FastIF: Scalable Influence Functions for Efficient Model Interpretation and Debuggingcited by 120
  11. 2018Understanding the Origins of Bias in Word Embeddingscited by 90
  12. 2019Analyzing Information Leakage of Updates to Natural Language Modelscited by 89
  13. 2019On Second-Order Group Influence Functions for Black-Box Predictionscited by 78
  14. 2020Coded Machine Unlearningcited by 35
  15. 2019SoK: Differential Privaciescited by 33
  16. 2018An Analysis of Neural Language Modeling at Multiple Scalescited by 31
  17. 2019Can You Trust This Prediction? Auditing Pointwise Reliability After Learningcited by 28
  18. 2020RelatIF: Identifying Explanatory Training Examples via Relative Influencecited by 28
  19. 2020Multi-Stage Influence Functioncited by 16
  20. 2018A scalable estimate of the extra-sample prediction error via approximate leave-one-outcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 25; 2024: 79; 2025: 71; 2026: 41.

223 papers.

Sort:

By area:

  1. 2023SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generationcited by 376
  2. 2022A Survey of Machine Unlearningcited by 298
  3. 2023Large Language Model Unlearningcited by 288
  4. 2024Rethinking Machine Unlearning for Large Language Modelscited by 247
  5. 2023Model Sparsity Can Simplify Machine Unlearningcited by 206
  6. 2023To Generate or Not? Safety-Driven Unlearned Diffusion Models Are Still Easy To Generate Unsafe Images ... For Nowcited by 204
  7. 2024Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearningcited by 160
  8. 2024SOUL: Unlocking the Power of Second-Order Optimization for LLM Unlearningcited by 104
  9. 2022Federated Unlearning: How to Efficiently Erase a Client in FL?cited by 90
  10. 2024What makes unlearning hard and what to do about itcited by 89
  11. 2023Machine Unlearning: Solutions and Challengescited by 85
  12. 2022Towards Adversarial Evaluations for Inexact Machine Unlearningcited by 82
38 more
  1. 2024LLM Unlearning via Loss Adjustment with Only Forget Datacited by 71
  2. 2024Scissorhands: Scrub Data Influence via Connection Sensitivity in Networkscited by 50
  3. 2024Machine Unlearning in Generative AI: A Surveycited by 49
  4. 2024Machine Unlearning for Image-to-Image Generative Modelscited by 47
  5. 2024UnlearnCanvas: Stylized Image Dataset for Enhanced Machine Unlearning Evaluation in Diffusion Modelscited by 47
  6. 2023Exploring the Landscape of Machine Unlearning: A Comprehensive Survey and Taxonomycited by 44
  7. 2024Challenging Forgets: Unveiling the Worst-Case Forget Sets in Machine Unlearningcited by 44
  8. 2023GNNDelete: A General Strategy for Unlearning in Graph Neural Networkscited by 43
  9. 2024Unified Gradient-Based Machine Unlearning with Remain Geometry Enhancementcited by 43
  10. 2022Evaluating Machine Unlearning via Epistemic Uncertaintycited by 39
  11. 2024LLM-PBE: Assessing Data Privacy in Large Language Modelscited by 38
  12. 2023Learn to Unlearn for Deep Neural Networks: Minimizing Unlearning Interference with Gradient Projectioncited by 37
  13. 2024Threats, Attacks, and Defenses in Machine Unlearning: A Surveycited by 34
  14. 2024Machine Unlearning: A Comprehensive Surveycited by 33
  15. 2024Revisiting Who's Harry Potter: Towards Targeted Unlearning from a Causal Intervention Perspectivecited by 33
  16. 2022Deep Regression Unlearningcited by 32
  17. 2024A Review on Machine Unlearningcited by 30
  18. 2023Certified Minimax Unlearning with Generalization Rates and Deletion Capacitycited by 29
  19. 2024Towards Certified Unlearning for Deep Neural Networkscited by 28
  20. 2022Repairing Neural Networks by Leaving the Right Past Behindcited by 27
  21. 2024On Effects of Steering Latent Representation for Large Language Model Unlearningcited by 27
  22. 2024Corrective Machine Unlearningcited by 26
  23. 2024WAGLE: Strategic Weight Attribution for Effective and Modular Unlearning in Large Language Modelscited by 25
  24. 2022Verifiable and Provably Secure Machine Unlearningcited by 24
  25. 2024Editable Concept Bottleneck Modelscited by 22
  26. 2024Score Forgetting Distillation: A Swift, Data-Free Method for Machine Unlearning in Diffusion Modelscited by 22
  27. 2023Breaking the Trilemma of Privacy, Utility, Efficiency via Controllable Machine Unlearningcited by 21
  28. 2023Deep Unlearning: Fast and Efficient Gradient-free Approach to Class Forgettingcited by 21
  29. 2024Graph Unlearning with Efficient Partial Retrainingcited by 20
  30. 2024Ethos: Rectifying Language Models in Orthogonal Parameter Spacecited by 20
  31. 2024Verification of Machine Unlearning is Fragilecited by 20
  32. 2024When Machine Unlearning Meets Retrieval-Augmented Generation (RAG): Keep Secret or Forget Knowledge?cited by 20
  33. 2024Towards Robust Evaluation of Unlearning in LLMs via Data Transformationscited by 19
  34. 2025Learning to Unlearn while Retaining: Combating Gradient Conflicts in Machine Unlearningcited by 19
  35. 2023Machine Unlearning: A Surveycited by 18
  36. 2023Zero-Knowledge Proof-based Verifiable Decentralized Machine Learning in Communication Network: A Comprehensive Surveycited by 18
  37. 2024Attribute-to-Delete: Machine Unlearning via Datamodel Matchingcited by 18
  38. 2024Second-Order Information Matters: Revisiting Machine Unlearning for Large Language Modelscited by 17

50 most cited of 223 All 223 (JSON)