# Evaluating Machine Unlearning via Epistemic Uncertainty

arXiv: https://arxiv.org/abs/2208.10836 · HTML: https://mlsanity.com/paper/2208.10836/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Placed in Machine unlearning, one of 12 directions in LLM safety and security.
- Builds on 23 of its 32 references that are in the map.
- First posted on arXiv 23 Aug 2022.

## Placed in

- LLM safety and security / [Machine unlearning](https://mlsanity.com/safety/machine-unlearning/)

## Builds on (most cited first, 23 of 23)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Machine Unlearning](https://mlsanity.com/paper/1912.03817/) (2019)
- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks](https://mlsanity.com/paper/1911.04933/) (2019)
- [Certified Data Removal from Machine Learning Models](https://mlsanity.com/paper/1911.03030/) (2019)
- [New insights and perspectives on the natural gradient method](https://mlsanity.com/paper/1412.1193/) (2014)
- [Remember What You Want to Forget: Algorithms for Machine Unlearning](https://mlsanity.com/paper/2103.03279/) (2021)
- [Amnesiac Machine Learning](https://mlsanity.com/paper/2010.10981/) (2020)
- [Descent-to-Delete: Gradient-Based Methods for Machine Unlearning](https://mlsanity.com/paper/2007.02923/) (2020)
- [Machine Unlearning of Features and Labels](https://mlsanity.com/paper/2108.11577/) (2021)
- [Adversarial Robustness Toolbox v1.0.0](https://mlsanity.com/paper/1807.01069/) (2018)
- [Adaptive Machine Unlearning](https://mlsanity.com/paper/2106.04378/) (2021)
- [Mixed-Privacy Forgetting in Deep Networks](https://mlsanity.com/paper/2012.13431/) (2020)
- [Machine Unlearning for Random Forests](https://mlsanity.com/paper/2009.05567/) (2020)
- [Machine Unlearning via Algorithmic Stability](https://mlsanity.com/paper/2102.13179/) (2021)
- [Federated Unlearning via Class-Discriminative Pruning](https://mlsanity.com/paper/2110.11794/) (2021)
- [Recommendation Unlearning](https://mlsanity.com/paper/2201.06820/) (2022)
- [Backdoor Defense with Machine Unlearning](https://mlsanity.com/paper/2201.09538/) (2022)
- [Coded Machine Unlearning](https://mlsanity.com/paper/2012.15721/) (2020)
- [Markov Chain Monte Carlo-Based Machine Unlearning: Unlearning What Needs to be Forgotten](https://mlsanity.com/paper/2202.13585/) (2022)
- [Knowledge Removal in Sampling-based Bayesian Inference](https://mlsanity.com/paper/2203.12964/) (2022)
- [Efficiently Maintaining Next Basket Recommendations under Additions and Deletions of Baskets and Items](https://mlsanity.com/paper/2201.13313/) (2022)

## Built on by (most cited first, 30 of 39)

- [SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation](https://mlsanity.com/paper/2310.12508/) (2023)
- [A Survey of Machine Unlearning](https://mlsanity.com/paper/2209.02299/) (2022)
- [Rethinking Machine Unlearning for Large Language Models](https://mlsanity.com/paper/2402.08787/) (2024)
- [Model Sparsity Can Simplify Machine Unlearning](https://mlsanity.com/paper/2304.04934/) (2023)
- [To Generate or Not? Safety-Driven Unlearned Diffusion Models Are Still Easy To Generate Unsafe Images ... For Now](https://mlsanity.com/paper/2310.11868/) (2023)
- [LLM Unlearning via Loss Adjustment with Only Forget Data](https://mlsanity.com/paper/2410.11143/) (2024)
- [Scissorhands: Scrub Data Influence via Connection Sensitivity in Networks](https://mlsanity.com/paper/2401.06187/) (2024)
- [Exploring the Landscape of Machine Unlearning: A Comprehensive Survey and Taxonomy](https://mlsanity.com/paper/2305.06360/) (2023)
- [UnlearnCanvas: Stylized Image Dataset for Enhanced Machine Unlearning Evaluation in Diffusion Models](https://mlsanity.com/paper/2402.11846/) (2024)
- [Challenging Forgets: Unveiling the Worst-Case Forget Sets in Machine Unlearning](https://mlsanity.com/paper/2403.07362/) (2024)
- [Machine Unlearning: A Comprehensive Survey](https://mlsanity.com/paper/2405.07406/) (2024)
- [WAGLE: Strategic Weight Attribution for Effective and Modular Unlearning in Large Language Models](https://mlsanity.com/paper/2410.17509/) (2024)
- [Score Forgetting Distillation: A Swift, Data-Free Method for Machine Unlearning in Diffusion Models](https://mlsanity.com/paper/2409.11219/) (2024)
- [Breaking the Trilemma of Privacy, Utility, Efficiency via Controllable Machine Unlearning](https://mlsanity.com/paper/2310.18574/) (2023)
- [Towards Robust Evaluation of Unlearning in LLMs via Data Transformations](https://mlsanity.com/paper/2411.15477/) (2024)
- [An Information Theoretic Evaluation Metric For Strong Unlearning](https://mlsanity.com/paper/2405.17878/) (2024)
- [Label Smoothing Improves Machine Unlearning](https://mlsanity.com/paper/2406.07698/) (2024)
- [Towards Reliable Forgetting: A Survey on Machine Unlearning Verification](https://mlsanity.com/paper/2506.15115/) (2025)
- [Go Beyond Your Means: Unlearning with Per-Sample Gradient Orthogonalization](https://mlsanity.com/paper/2503.02312/) (2025)
- [Towards Lifecycle Unlearning Commitment Management: Measuring Sample-level Approximate Unlearning Completeness](https://mlsanity.com/paper/2403.12830/) (2024)
- [Forget Vectors at Play: Universal Input Perturbations Driving Machine Unlearning in Image Classification](https://mlsanity.com/paper/2412.16780/) (2024)
- [Targeted Forgetting of Image Subgroups in CLIP Models](https://mlsanity.com/paper/2506.03117/) (2025)
- [Do Not Mimic My Voice: Speaker Identity Unlearning for Zero-Shot Text-to-Speech](https://mlsanity.com/paper/2507.20140/) (2025)
- [A Reliable Cryptographic Framework for Empirical Machine Unlearning Evaluation](https://mlsanity.com/paper/2404.11577/) (2024)
- [Understanding Fine-tuning in Approximate Unlearning: A Theoretical Perspective](https://mlsanity.com/paper/2410.03833/) (2024)
- [Efficient Utility-Preserving Machine Unlearning with Implicit Gradient Surgery](https://mlsanity.com/paper/2510.22124/) (2025)
- [On the Impact of Uncertainty and Calibration on Likelihood-Ratio Membership Inference Attacks](https://mlsanity.com/paper/2402.10686/) (2024)
- [Unlearning Comparator: A Visual Analytics System for Comparative Evaluation of Machine Unlearning Methods](https://mlsanity.com/paper/2508.12730/) (2025)
- [Tackling Fake Forgetting through Uncertainty Quantification](https://mlsanity.com/paper/2501.19403/) (2025)
- [Discriminative Adversarial Unlearning](https://mlsanity.com/paper/2402.06864/) (2024)

## Abstract (arXiv)

There has been a growing interest in Machine Unlearning recently, primarily due to legal requirements such as the General Data Protection Regulation (GDPR) and the California Consumer Privacy Act. Thus, multiple approaches were presented to remove the influence of specific target data points from a trained model. However, when evaluating the success of unlearning, current approaches either use adversarial attacks or compare their results to the optimal solution, which usually incorporates retraining from scratch. We argue that both ways are insufficient in practice. In this work, we present an evaluation metric for Machine Unlearning algorithms based on epistemic uncertainty. This is the first definition of a general evaluation metric for Machine Unlearning to our best knowledge.