# Towards Machine Unlearning Benchmarks: Forgetting the Personal Identities in Facial Recognition Systems

arXiv: https://arxiv.org/abs/2311.02240 · HTML: https://mlsanity.com/paper/2311.02240/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Machine unlearning, one of 12 directions in LLM safety and security.
- Builds on 15 of its 25 references that are in the map.
- First posted on arXiv 3 Nov 2023.

## Placed in

- LLM safety and security / [Machine unlearning](https://mlsanity.com/safety/machine-unlearning/)

## Builds on (most cited first, 15 of 15)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Extracting Training Data from Large Language Models](https://mlsanity.com/paper/2012.07805/) (2020)
- [Remember What You Want to Forget: Algorithms for Machine Unlearning](https://mlsanity.com/paper/2103.03279/) (2021)
- [Descent-to-Delete: Gradient-Based Methods for Machine Unlearning](https://mlsanity.com/paper/2007.02923/) (2020)
- [Unrolling SGD: Understanding Factors Influencing Machine Unlearning](https://mlsanity.com/paper/2109.13398/) (2021)
- [Forgetting Outside the Box: Scrubbing Deep Networks of Information Accessible from Input-Output Observations](https://mlsanity.com/paper/2003.02960/) (2020)
- [Adaptive Machine Unlearning](https://mlsanity.com/paper/2106.04378/) (2021)
- [Machine Unlearning via Algorithmic Stability](https://mlsanity.com/paper/2102.13179/) (2021)
- [Bag of Tricks for Training Data Extraction from Language Models](https://mlsanity.com/paper/2302.04460/) (2023)

## Built on by (most cited first, 28 of 28)

- [The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning](https://mlsanity.com/paper/2403.03218/) (2024)
- [UnlearnCanvas: Stylized Image Dataset for Enhanced Machine Unlearning Evaluation in Diffusion Models](https://mlsanity.com/paper/2402.11846/) (2024)
- [Erasing Conceptual Knowledge from Language Models](https://mlsanity.com/paper/2410.02760/) (2024)
- [LoTUS: Large-Scale Machine Unlearning with a Taste of Uncertainty](https://mlsanity.com/paper/2503.18314/) (2025)
- [Ferrari: Federated Feature Unlearning via Optimizing Feature Sensitivity](https://mlsanity.com/paper/2405.17462/) (2024)
- [Membership Inference Attacks as Privacy Tools: Reliability, Disparity and Ensemble](https://mlsanity.com/paper/2506.13972/) (2025)
- [Machine Unlearning under Retain-Forget Entanglement](https://mlsanity.com/paper/2603.26569/) (2026)
- [Gone but Not Forgotten: Improved Benchmarks for Machine Unlearning](https://mlsanity.com/paper/2405.19211/) (2024)
- [Distribution-Level Feature Distancing for Machine Unlearning: Towards a Better Trade-off Between Model Utility and Forgetting](https://mlsanity.com/paper/2409.14747/) (2024)
- [CURE4Rec: A Benchmark for Recommendation Unlearning with Deeper Influence](https://mlsanity.com/paper/2408.14393/) (2024)
- [Attack and Reset for Unlearning: Exploiting Adversarial Noise toward Machine Unlearning through Parameter Re-initialization](https://mlsanity.com/paper/2401.08998/) (2024)
- [Towards Privacy-Guaranteed Label Unlearning in Vertical Federated Learning: Few-Shot Forgetting without Disclosure](https://mlsanity.com/paper/2410.10922/) (2024)
- ["Alexa, can you forget me?" Machine Unlearning Benchmark in Spoken Language Understanding](https://mlsanity.com/paper/2505.15700/) (2025)
- [Unlearning Personal Data from a Single Image](https://mlsanity.com/paper/2407.12069/) (2024)
- [Zero-shot Class Unlearning via Layer-wise Relevance Analysis and Neuronal Path Perturbation](https://mlsanity.com/paper/2410.23693/) (2024)
- [An Illusion of Unlearning? Assessing Machine Unlearning Through Internal Representations](https://mlsanity.com/paper/2604.08271/) (2026)
- [Evaluating the Defense Potential of Machine Unlearning against Membership Inference Attacks](https://mlsanity.com/paper/2508.16150/) (2025)
- [ViT-MUL: A Baseline Study on Recent Machine Unlearning Methods Applied to Vision Transformers](https://mlsanity.com/paper/2403.09681/) (2024)
- [ToFU: Transforming How Federated Learning Systems Forget User Data](https://mlsanity.com/paper/2509.15861/) (2025)
- [CURE: Centroid-guided Unsupervised Representation Erasure for Facial Recognition Systems](https://mlsanity.com/paper/2509.19562/) (2025)
- [FAME: Fictional Actors for Multilingual Erasure](https://mlsanity.com/paper/2512.15235/) (2025)
- [Robust Continual Unlearning against Knowledge Erosion and Forgetting Reversal](https://mlsanity.com/paper/2604.19108/) (2026)
- [Grokked Models are Better Unlearners](https://mlsanity.com/paper/2512.03437/) (2025)
- [Classification-Head Bias in Class-Level Machine Unlearning: Diagnosis, Mitigation, and Evaluation](https://mlsanity.com/paper/2605.08730/) (2026)
- [Interference-Aware Multi-Task Unlearning](https://mlsanity.com/paper/2605.19042/) (2026)
- [REMEDI: A Benchmark for Retention and Unlearning Evaluation in Multi-label Clinical Disease Inference](https://mlsanity.com/paper/2606.07141/) (2026)
- [Cross-Domain Generalization in Machine Unlearning via Label-Conditioned Energy Magnitude Regularization](https://mlsanity.com/paper/2608.17942/) (2026)
- [Damnatio Memoriae: Adversarially and Selectively Forgetting Identities in the Embedding Space of Face Recognition Models](https://mlsanity.com/paper/2609.27115/) (2026)

## Abstract (arXiv)

Machine unlearning is a crucial tool for enabling a classification model to forget specific data that are used in the training time. Recently, various studies have presented machine unlearning algorithms and evaluated their methods on several datasets. However, most of the current machine unlearning algorithms have been evaluated solely on traditional computer vision datasets such as CIFAR-10, MNIST, and SVHN. Furthermore, previous studies generally evaluate the unlearning methods in the class-unlearning setup. Most previous work first trains the classification models and then evaluates the machine unlearning performance of machine unlearning algorithms by forgetting selected image classes (categories) in the experiments. Unfortunately, these class-unlearning settings might not generalize to real-world scenarios. In this work, we propose a machine unlearning setting that aims to unlearn specific instance that contains personal privacy (identity) while maintaining the original task of a given model. Specifically, we propose two machine unlearning benchmark datasets, MUFAC and MUCAC, that are greatly useful to evaluate the performance and robustness of a machine unlearning algorithm. In our benchmark datasets, the original model performs facial feature recognition tasks: face age estimation (multi-class classification) and facial attribute classification (binary class classification), where a class does not depend on any single target subject (personal identity), which can be a realistic setting. Moreover, we also report the performance of the state-of-the-art machine unlearning methods on our proposed benchmark datasets. All the datasets, source codes, and trained models are publicly available at https://github.com/ndb796/MachineUnlearning.