# ICU-Bench:Benchmarking Continual Unlearning in Multimodal Large Language Models

arXiv: https://arxiv.org/abs/2605.05938 · HTML: https://mlsanity.com/paper/2605.05938/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 31 of its 34 references that are in the map.
- First posted on arXiv 7 May 2026.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM jailbreaks and adversarial robustness](https://mlsanity.com/mllm/mllm-jailbreaks-and-adversarial-robustness/)
- LLM safety and security / [Machine unlearning](https://mlsanity.com/safety/machine-unlearning/)

## Builds on (most cited first, 30 of 31)

- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [mPLUG-Owl: Modularization Empowers Large Language Models with Multimodality](https://mlsanity.com/paper/2304.14178/) (2023)
- [TOFU: A Task of Fictitious Unlearning for LLMs](https://mlsanity.com/paper/2401.06121/) (2024)
- [Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearning](https://mlsanity.com/paper/2404.05868/) (2024)
- [Knowledge Unlearning for Mitigating Privacy Risks in Language Models](https://mlsanity.com/paper/2210.01504/) (2022)
- [A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Ugly](https://mlsanity.com/paper/2312.02003/) (2023)
- [LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models](https://mlsanity.com/paper/2407.12772/) (2024)
- [Large Language Model Unlearning](https://mlsanity.com/paper/2310.10683/) (2023)
- [Unrolling SGD: Understanding Factors Influencing Machine Unlearning](https://mlsanity.com/paper/2109.13398/) (2021)
- [MUSE: Machine Unlearning Six-Way Evaluation for Language Models](https://mlsanity.com/paper/2407.06460/) (2024)
- [Rethinking Machine Unlearning for Large Language Models](https://mlsanity.com/paper/2402.08787/) (2024)
- [Continual Learning and Private Unlearning](https://mlsanity.com/paper/2203.12817/) (2022)
- [Variational Bayesian Unlearning](https://mlsanity.com/paper/2010.12883/) (2020)
- [Protecting Privacy in Multimodal Large Language Models with MLLMU-Bench](https://mlsanity.com/paper/2410.22108/) (2024)
- [On Large Language Model Continual Unlearning](https://mlsanity.com/paper/2407.10223/) (2024)
- [Single Image Unlearning: Efficient Machine Unlearning in Multimodal Large Language Models](https://mlsanity.com/paper/2405.12523/) (2024)
- [CLEAR: Character Unlearning in Textual and Visual Modalities](https://mlsanity.com/paper/2410.18057/) (2024)
- [Knowledge Unlearning for LLMs: Tasks, Methods, and Challenges](https://mlsanity.com/paper/2311.15766/) (2023)
- [MMUnlearner: Reformulating Multimodal Machine Unlearning in the Era of Multimodal Large Language Models](https://mlsanity.com/paper/2502.11051/) (2025)
- [WAGLE: Strategic Weight Attribution for Effective and Modular Unlearning in Large Language Models](https://mlsanity.com/paper/2410.17509/) (2024)
- [Modality-Aware Neuron Pruning for Unlearning in Multimodal Large Language Models](https://mlsanity.com/paper/2502.15910/) (2025)
- [Cross-Modal Safety Alignment: Is textual unlearning all you need?](https://mlsanity.com/paper/2406.02575/) (2024)
- [EFUF: Efficient Fine-grained Unlearning Framework for Mitigating Hallucinations in Multimodal Large Language Models](https://mlsanity.com/paper/2402.09801/) (2024)
- [MU-Bench: A Multitask Multimodal Benchmark for Machine Unlearning](https://mlsanity.com/paper/2406.14796/) (2024)
- [VideoEraser: Concept Erasure in Text-to-Video Diffusion Models](https://mlsanity.com/paper/2508.15314/) (2025)
- [Yuan: Yielding Unblemished Aesthetics Through A Unified Network for Visual Imperfections Removal in Generated Images](https://mlsanity.com/paper/2501.08505/) (2025)
- [ForgetMe: Evaluating Selective Forgetting in Generative Models](https://mlsanity.com/paper/2504.12574/) (2025)
- [PEBench: A Fictitious Dataset to Benchmark Machine Unlearning for Multimodal Large Language Models](https://mlsanity.com/paper/2503.12545/) (2025)

## Built on by (most cited first, 1 of 1)

- [Beyond Cross-Lingual Transfer: Benchmarking Propagation Boundaries in Multilingual LLM Unlearning](https://mlsanity.com/paper/2609.05976/) (2026)

## Abstract (arXiv)

Privacy deletion requests often arrive sequentially, creating a continual unlearning challenge for deployed multimodal large language models (MLLMs). However, existing benchmarks mainly focus on static or short-sequence settings, offering limited support for evaluating continual privacy deletion on privacy-critical documents. To bridge this gap, we introduce ICU-Bench, an Identity-centric Continual Unlearning benchmark (pronounced ``I see you'') for privacy-critical multimodal documents. ICU-Bench contains 1,000 synthetic privacy-sensitive profiles from medical reports and labor contracts, comprising 9,500 document images, 16,000 question-answer pairs, and 100 sequential forget tasks. We further introduce history-aware evaluation protocols and sequence-aware metrics to assess current forgetting, historical forgetting preservation, retained utility, and stability throughout the unlearning sequence. Experiments with representative unlearning methods on two MLLMs reveal that methods effective on current targets often fail to preserve earlier forgetting or retained capabilities over long sequences. Some methods further obtain low forget accuracy through severe model degradation. These results expose long-horizon failure modes overlooked by conventional evaluations and highlight the need for multimodal unlearning methods explicitly designed for continual privacy deletion. Our code and benchmark are publicly available at: https://github.com/AstorYH/ICU-Bench.