# LetheViT: Selective Machine Unlearning for Vision Transformers via Attention-Guided Contrastive Learning

arXiv: https://arxiv.org/abs/2508.01569 · HTML: https://mlsanity.com/paper/2508.01569/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Machine unlearning, one of 12 directions in LLM safety and security.
- Builds on 25 of its 31 references that are in the map.
- First posted on arXiv 3 Aug 2025.

## Placed in

- LLM safety and security / [Machine unlearning](https://mlsanity.com/safety/machine-unlearning/)

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networks](https://mlsanity.com/paper/1911.04933/) (2019)
- [Making AI Forget You: Data Deletion in Machine Learning](https://mlsanity.com/paper/1907.05012/) (2019)
- [SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation](https://mlsanity.com/paper/2310.12508/) (2023)
- [Remember What You Want to Forget: Algorithms for Machine Unlearning](https://mlsanity.com/paper/2103.03279/) (2021)
- [Amnesiac Machine Learning](https://mlsanity.com/paper/2010.10981/) (2020)
- [Unrolling SGD: Understanding Factors Influencing Machine Unlearning](https://mlsanity.com/paper/2109.13398/) (2021)
- [Approximate Data Deletion from Machine Learning Models](https://mlsanity.com/paper/2002.10077/) (2020)
- [Machine Unlearning of Features and Labels](https://mlsanity.com/paper/2108.11577/) (2021)
- [Model Sparsity Can Simplify Machine Unlearning](https://mlsanity.com/paper/2304.04934/) (2023)
- [Simplicity Prevails: Rethinking Negative Preference Optimization for LLM Unlearning](https://mlsanity.com/paper/2410.07163/) (2024)
- [FRAMU: Attention-based Machine Unlearning using Federated Reinforcement Learning](https://mlsanity.com/paper/2309.10283/) (2023)
- [Robust Machine Unlearning for Quantized Neural Networks via Adaptive Gradient Reweighting with Similar Labels](https://mlsanity.com/paper/2503.13917/) (2025)
- [NOVO: Unlearning-Compliant Vision Transformers](https://mlsanity.com/paper/2507.03281/) (2025)
- [DFQ-ViT: Data-Free Quantization for Vision Transformers without Fine-tuning](https://mlsanity.com/paper/2507.14481/) (2025)
- [An Empirical Evaluation of the Implementation of the California Consumer Privacy Act (CCPA)](https://mlsanity.com/paper/2205.09897/) (2022)

## Built on by (most cited first, 3 of 3)

- [TOUR: A Trajectory-Level Unlearning Benchmark for Offline Reinforcement Learning](https://mlsanity.com/paper/2607.21111/) (2026)
- [Benchmarking Unlearning for Vision Transformers](https://mlsanity.com/paper/2602.20114/) (2026)
- [Interference-Aware Multi-Task Unlearning](https://mlsanity.com/paper/2605.19042/) (2026)

## Abstract (arXiv)

Vision Transformers (ViTs) have revolutionized computer vision tasks with their exceptional performance. However, the introduction of privacy regulations such as GDPR and CCPA has brought new challenges to them. These laws grant users the right to withdraw their data, necessitating not only the deletion of data but also the complete removal of its influence from trained models. Machine unlearning emerges as a critical solution, with exact unlearning being computationally prohibitive and approximate methods offering a more practical approach. This work addresses the particularly challenging scenario of random data forgetting in ViTs, where the model must forget specific samples while retaining others, even within the same class. We first reveal the core characteristics of ViTs through selective masking experiments: when high-attention areas are masked, the model retains its recognition capability but significantly weakens its memorization ability. Based on the above insights, we propose LetheViT, a contrastive unlearning method tailored for ViTs. LetheViT uses masked image inputs to generate positive logits and original image inputs to generate negative logits, guiding the model to forget specific details while retaining the general cl category outlines. Experimental results demonstrate that LetheViT achieves state-of-the-art performance, effectively balancing privacy compliance with model efficacy.