# Suppression Is Not Forgetting: Residual Recoverability in Visual Concept Unlearning for VLMs

arXiv: https://arxiv.org/abs/2604.03114 · HTML: https://mlsanity.com/paper/2604.03114/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Machine unlearning, one of 12 directions in LLM safety and security.
- Builds on 26 of its 29 references that are in the map.
- First posted on arXiv 3 Apr 2026.

## Placed in

- LLM safety and security / [Machine unlearning](https://mlsanity.com/safety/machine-unlearning/)

## Builds on (most cited first, 26 of 26)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Models](https://mlsanity.com/paper/2504.10479/) (2025)
- [Machine Unlearning](https://mlsanity.com/paper/1912.03817/) (2019)
- [TOFU: A Task of Fictitious Unlearning for LLMs](https://mlsanity.com/paper/2401.06121/) (2024)
- [The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearning](https://mlsanity.com/paper/2403.03218/) (2024)
- [Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning](https://mlsanity.com/paper/2404.05868/) (2024)
- [Knowledge Unlearning for Mitigating Privacy Risks in Language Models](https://mlsanity.com/paper/2210.01504/) (2022)
- [AID: A Benchmark Dataset for Performance Evaluation of Aerial Scene Classification](https://mlsanity.com/paper/1608.05167/) (2016)
- [A Survey of Machine Unlearning](https://mlsanity.com/paper/2209.02299/) (2022)
- [MUSE: Machine Unlearning Six-Way Evaluation for Language Models](https://mlsanity.com/paper/2407.06460/) (2024)
- [In-Context Unlearning: Language Models as Few Shot Unlearners](https://mlsanity.com/paper/2310.07579/) (2023)
- [Guardrail Baselines for Unlearning in LLMs](https://mlsanity.com/paper/2403.03329/) (2024)
- [RWKU: Benchmarking Real-World Knowledge Unlearning for Large Language Models](https://mlsanity.com/paper/2406.10890/) (2024)
- [Protecting Privacy in Multimodal Large Language Models with MLLMU-Bench](https://mlsanity.com/paper/2410.22108/) (2024)
- [Single Image Unlearning: Efficient Machine Unlearning in Multimodal Large Language Models](https://mlsanity.com/paper/2405.12523/) (2024)
- [Soft Prompting for Unlearning in Large Language Models](https://mlsanity.com/paper/2406.12038/) (2024)
- [CLEAR: Character Unlearning in Textual and Visual Modalities](https://mlsanity.com/paper/2410.18057/) (2024)
- [Benchmarking Vision Language Model Unlearning via Fictitious Facial Identity Dataset](https://mlsanity.com/paper/2411.03554/) (2024)
- [Does Unlearning Truly Unlearn? A Black Box Evaluation of LLM Unlearning Methods](https://mlsanity.com/paper/2411.12103/) (2024)
- [Better Hit the Nail on the Head than Beat around the Bush: Removing Protected Attributes with a Single Projection](https://mlsanity.com/paper/2212.04273/) (2022)
- [MLLMEraser: Achieving Test-Time Unlearning in Multimodal Large Language Models through Activation Steering](https://mlsanity.com/paper/2510.04217/) (2025)
- [Answer When Needed, Forget When Not: Language Models Pretend to Forget via In-Context Knowledge Unlearning](https://mlsanity.com/paper/2410.00382/) (2024)
- [Knowledge Vector Weakening: Efficient Training-free Unlearning for Large Vision-Language Models](https://mlsanity.com/paper/2601.21794/) (2026)
- [AUVIC: Adversarial Unlearning of Visual Concepts for Multi-modal Large Language Models](https://mlsanity.com/paper/2511.11299/) (2025)

## Built on by (most cited first, 1 of 1)

- [LLM Unlearning for Cyber Defense: A Survey on Methods, Challenges, and Emerging Threats](https://mlsanity.com/paper/2607.16227/) (2026)

## Abstract (arXiv)

Vision-language models (VLMs) may need to forget visual concepts after deployment because of privacy, copyright, licensing, safety, or policy changes. Conventional machine unlearning modifies model parameters, which may be costly or inaccessible for API-only models. Prompt-based suppression offers a training-free alternative, but does it make a concept inaccessible or merely change the model's answer? We investigate this question in off-the-shelf VLMs. Our visually grounded, multi-probe evaluation first verifies that a model recognizes each concept from the image, then tests its recoverability through multiple-choice, short-answer, and indirect queries. Across objects, scenes, and identities, prompt suppression reduces short-answer recall for some concepts while leaving multiple-choice and indirect performance largely unchanged. Explicitly listing the concepts to suppress often increases short-answer recall, suggesting that the list itself cues the answer. Beyond prompting, decoding constraints, representation editing, and parameter updates can suppress particular responses while the same concept remains detectable through other queries. A model may stop naming a visual concept yet still identify or use it when queried differently.