# Label-Agnostic Forgetting: A Supervision-Free Unlearning in Deep Models

arXiv: https://arxiv.org/abs/2404.00506 · HTML: https://mlsanity.com/paper/2404.00506/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Machine unlearning, one of 12 directions in LLM safety and security.
- Builds on 21 of its 34 references that are in the map.
- First posted on arXiv 31 Mar 2024.

## Placed in

- LLM safety and security / [Machine unlearning](https://mlsanity.com/safety/machine-unlearning/)

## Builds on (most cited first, 21 of 21)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Identity Mappings in Deep Residual Networks](https://mlsanity.com/paper/1603.05027/) (2016)
- [Variational Graph Auto-Encoders](https://mlsanity.com/paper/1611.07308/) (2016)
- [Machine Unlearning](https://mlsanity.com/paper/1912.03817/) (2019)
- [Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks](https://mlsanity.com/paper/1911.04933/) (2019)
- [Amnesiac Machine Learning](https://mlsanity.com/paper/2010.10981/) (2020)
- [A Survey of Machine Unlearning](https://mlsanity.com/paper/2209.02299/) (2022)
- [Variational Autoencoders for Collaborative Filtering](https://mlsanity.com/paper/1802.05814/) (2018)
- [Unrolling SGD: Understanding Factors Influencing Machine Unlearning](https://mlsanity.com/paper/2109.13398/) (2021)
- [Towards Unbounded Machine Unlearning](https://mlsanity.com/paper/2302.09880/) (2023)
- [A Survey on Deep Semi-supervised Learning](https://mlsanity.com/paper/2103.00550/) (2021)
- [Fast Yet Effective Machine Unlearning](https://mlsanity.com/paper/2111.08947/) (2021)
- [When Machine Unlearning Jeopardizes Privacy](https://mlsanity.com/paper/2005.02205/) (2020)
- [Can Bad Teaching Induce Forgetting? Unlearning in Deep Networks using an Incompetent Teacher](https://mlsanity.com/paper/2205.08096/) (2022)
- [Forgetting Outside the Box: Scrubbing Deep Networks of Information Accessible from Input-Output Observations](https://mlsanity.com/paper/2003.02960/) (2020)
- [Boundary Unlearning](https://mlsanity.com/paper/2303.11570/) (2023)
- [Variational Bayesian Unlearning](https://mlsanity.com/paper/2010.12883/) (2020)
- [Hidden Poison: Machine Unlearning Enables Camouflaged Poisoning Attacks](https://mlsanity.com/paper/2212.10717/) (2022)

## Built on by (most cited first, 18 of 18)

- [Decoupled Distillation to Erase: A General Unlearning Method for Any Class-centric Tasks](https://mlsanity.com/paper/2503.23751/) (2025)
- [Learning to Unlearn for Robust Machine Unlearning](https://mlsanity.com/paper/2407.10494/) (2024)
- [An Information Theoretic Evaluation Metric For Strong Unlearning](https://mlsanity.com/paper/2405.17878/) (2024)
- [Distribution-Level Feature Distancing for Machine Unlearning: Towards a Better Trade-off Between Model Utility and Forgetting](https://mlsanity.com/paper/2409.14747/) (2024)
- [MUBox: A Critical Evaluation Framework of Deep Machine Unlearning](https://mlsanity.com/paper/2505.08576/) (2025)
- [Understanding Fine-tuning in Approximate Unlearning: A Theoretical Perspective](https://mlsanity.com/paper/2410.03833/) (2024)
- [Unlearning Personal Data from a Single Image](https://mlsanity.com/paper/2407.12069/) (2024)
- [Tackling Fake Forgetting through Uncertainty Quantification](https://mlsanity.com/paper/2501.19403/) (2025)
- [Group-robust Machine Unlearning](https://mlsanity.com/paper/2503.09330/) (2025)
- [Models Designed to Forget: Machine Unlearning via Key Deletion](https://mlsanity.com/paper/2603.15033/) (2026)
- [Ready2Unlearn: A Learning-Time Approach for Preparing Models with Future Unlearning Readiness](https://mlsanity.com/paper/2505.10845/) (2025)
- [NOVO: Unlearning-Compliant Vision Transformers](https://mlsanity.com/paper/2507.03281/) (2025)
- [Reveal and Release: Iterative LLM Unlearning with Self-generated Data](https://mlsanity.com/paper/2509.14624/) (2025)
- [MUC: Machine Unlearning for Contrastive Learning with Black-box Evaluation](https://mlsanity.com/paper/2406.03603/) (2024)
- [Unlearning Evaluation through Subset Statistical Independence](https://mlsanity.com/paper/2603.00587/) (2026)
- [Machine Unlearning for Streaming Forgetting](https://mlsanity.com/paper/2507.15280/) (2025)
- [Machine Unlearning of Traffic State Estimation and Prediction](https://mlsanity.com/paper/2507.17984/) (2025)
- [Why Some Models Resist Unlearning: A Linear Stability Perspective](https://mlsanity.com/paper/2602.02986/) (2026)

## Abstract (arXiv)

Machine unlearning aims to remove information derived from forgotten data while preserving that of the remaining dataset in a well-trained model. With the increasing emphasis on data privacy, several approaches to machine unlearning have emerged. However, these methods typically rely on complete supervision throughout the unlearning process. Unfortunately, obtaining such supervision, whether for the forgetting or remaining data, can be impractical due to the substantial cost associated with annotating real-world datasets. This challenge prompts us to propose a supervision-free unlearning approach that operates without the need for labels during the unlearning process. Specifically, we introduce a variational approach to approximate the distribution of representations for the remaining data. Leveraging this approximation, we adapt the original model to eliminate information from the forgotten data at the representation level. To further address the issue of lacking supervision information, which hinders alignment with ground truth, we introduce a contrastive loss to facilitate the matching of representations between the remaining data and those of the original model, thus preserving predictive performance. Experimental results across various unlearning tasks demonstrate the effectiveness of our proposed method, Label-Agnostic Forgetting (LAF) without using any labels, which achieves comparable performance to state-of-the-art methods that rely on full supervision information. Furthermore, our approach excels in semi-supervised scenarios, leveraging limited supervision information to outperform fully supervised baselines. This work not only showcases the viability of supervision-free unlearning in deep models but also opens up a new possibility for future research in unlearning at the representation level.