mlsanityData to 7 Oct 2026

On the Necessity of Auditable Algorithmic Definitions for Machine Unlearning

arXiv 2110.11891 ↗First posted 22 Oct 2021cs.LG cs.AI cs.CRcited by 164

On the map: LLM safety and security → Machine unlearning

Abstract (arXiv)

Machine unlearning, i.e. having a model forget about some of its training data, has become increasingly more important as privacy legislation promotes variants of the right-to-be-forgotten. In the context of deep learning, approaches for machine unlearning are broadly categorized into two classes: exact unlearning methods, where an entity has formally removed the data point's impact on the model by retraining the model from scratch, and approximate unlearning, where an entity approximates the model parameters one would obtain by exact unlearning to save on compute costs. In this paper, we first show that the definition that underlies approximate unlearning, which seeks to prove the approximately unlearned model is close to an exactly retrained model, is incorrect because one can obtain the same model using different datasets. Thus one could unlearn without modifying the model at all. We then turn to exact unlearning approaches and ask how to verify their claims of unlearning. Our results show that even for a given training trajectory one cannot formally prove the absence of certain data points used during training. We thus conclude that unlearning is only well-defined at the algorithmic level, where an entity's only possible auditable claim to unlearning is that they used a particular algorithm designed to allow for external scrutiny during an audit.

Key facts. Cited by 164 papers in the map (data through 7 Oct 2026). Placed in Machine unlearning, one of 12 directions in LLM safety and security. Builds on 18 of its 26 references that are in the map. First posted on arXiv 22 Oct 2021.

Builds on

18 of 26 references.

Closest ancestors · 3

Same direction, newest first

  1. 2021Unrolling SGD: Understanding Factors Influencing Machine Unlearning27 Sep
  2. 2021Remember What You Want to Forget: Algorithms for Machine Unlearning4 Mar
  3. 2021Machine Unlearning via Algorithmic Stability25 Feb

Other references · 15

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  4. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  5. 2019Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networkscited by 599
  6. 2019Certified Data Removal from Machine Learning Modelscited by 512
  7. 2019Does Learning Require Memorization? A Short Tale about a Long Tailcited by 452
  8. 2019Making AI Forget You: Data Deletion in Machine Learningcited by 447
  9. 2020Amnesiac Machine Learningcited by 305
  10. 2020Forgetting Outside the Box: Scrubbing Deep Networks of Information Accessible from Input-Output Observationscited by 171
  11. 2020DeltaGrad: Rapid retraining of machine learning modelscited by 143
  12. 2020Machine Unlearning: Linear Filtration for Logit-based Classifierscited by 89
3 more
  1. 2021Proof-of-Learning: Definitions and Practicecited by 81
  2. 2021Manipulating SGD with Data Ordering Attackscited by 73
  3. 2021SoK: Machine Learning Governancecited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 11; 2023: 20; 2024: 53; 2025: 48; 2026: 30.

164 papers.

Sort:

By area:

  1. 2024TOFU: A Task of Fictitious Unlearning for LLMscited by 507
  2. 2023SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generationcited by 376
  3. 2022A Survey of Machine Unlearningcited by 298
  4. 2023Towards Unbounded Machine Unlearningcited by 261
  5. 2024Rethinking Machine Unlearning for Large Language Modelscited by 247
  6. 2021Fast Yet Effective Machine Unlearningcited by 209
  7. 2023Model Sparsity Can Simplify Machine Unlearningcited by 206
  8. 2023To Generate or Not? Safety-Driven Unlearned Diffusion Models Are Still Easy To Generate Unsafe Images ... For Nowcited by 204
  9. 2022Can Bad Teaching Induce Forgetting? Unlearning in Deep Networks using an Incompetent Teachercited by 189
  10. 2022The Right to be Forgotten in Federated Learning: An Efficient Realization with Rapid Retrainingcited by 109
  11. 2024AutoAttacker: A Large Language Model Guided System to Implement Automatic Cyber-attackscited by 94
  12. 2023Machine Unlearning: Solutions and Challengescited by 85
38 more
  1. 2022Towards Adversarial Evaluations for Inexact Machine Unlearningcited by 82
  2. 2024Inexact Unlearning Needs More Careful Evaluations to Avoid a False Sense of Privacycited by 78
  3. 2024UnUnlearning: Unlearning is not sufficient for content regulation in advanced generative AIcited by 46
  4. 2023Exploring the Landscape of Machine Unlearning: A Comprehensive Survey and Taxonomycited by 44
  5. 2024Challenging Forgets: Unveiling the Worst-Case Forget Sets in Machine Unlearningcited by 44
  6. 2023GNNDelete: A General Strategy for Unlearning in Graph Neural Networkscited by 43
  7. 2024Unified Gradient-Based Machine Unlearning with Remain Geometry Enhancementcited by 43
  8. 2022Algorithms that Approximate Data Removal: New Results and Limitationscited by 39
  9. 2023Knowledge Unlearning for LLMs: Tasks, Methods, and Challengescited by 37
  10. 2024Fast Exact Unlearning for In-Context Learning Data for LLMscited by 33
  11. 2024Machine Unlearning: A Comprehensive Surveycited by 33
  12. 2024A Review on Machine Unlearningcited by 30
  13. 2025Decoupled Distillation to Erase: A General Unlearning Method for Any Class-centric Taskscited by 28
  14. 2024Corrective Machine Unlearningcited by 26
  15. 2024Towards Scalable Exact Machine Unlearning Using Parameter-Efficient Fine-Tuningcited by 25
  16. 2024WAGLE: Strategic Weight Attribution for Effective and Modular Unlearning in Large Language Modelscited by 25
  17. 2025Certified Unlearning for Neural Networkscited by 25
  18. 2022Verifiable and Provably Secure Machine Unlearningcited by 24
  19. 2023Tools for Verifying Neural Models' Training Datacited by 21
  20. 2022Selective Prediction via Training Dynamicscited by 20
  21. 2023Efficiently Forgetting What You Have Learned in Graph Representation Learning via Projectioncited by 20
  22. 2023MultiDelete for Multimodal Machine Unlearningcited by 20
  23. 2024Graph Unlearning with Efficient Partial Retrainingcited by 20
  24. 2024Verification of Machine Unlearning is Fragilecited by 20
  25. 2025Language Models May Verbatim Complete Text They Were Not Explicitly Trained Oncited by 20
  26. 2023Machine Unlearning: A Surveycited by 18
  27. 2024Attribute-to-Delete: Machine Unlearning via Datamodel Matchingcited by 18
  28. 2024MUNBa: Machine Unlearning via Nash Bargainingcited by 17
  29. 2025LLM Unlearning with LLM Beliefscited by 17
  30. 2023Reinforcement Unlearningcited by 16
  31. 2024Adversarial Machine Unlearningcited by 16
  32. 2024MU-Bench: A Multitask Multimodal Benchmark for Machine Unlearningcited by 16
  33. 2021Forget-SVGD: Particle-Based Bayesian Federated Unlearningcited by 14
  34. 2024Contrastive Unlearning: A Contrastive Approach to Machine Unlearningcited by 14
  35. 2022An Introduction to Machine Unlearningcited by 13
  36. 2022Proof-of-Learning is Currently More Broken Than You Thinkcited by 12
  37. 2023Netflix and Forget: Efficient and Exact Machine Unlearning from Bi-linear Recommendationscited by 12
  38. 2023A Survey of Graph Unlearningcited by 12

50 most cited of 164 All 164 (JSON)