mlsanityData to 7 Oct 2026

ICU-Bench:Benchmarking Continual Unlearning in Multimodal Large Language Models

arXiv 2605.05938 ↗First posted 7 May 2026cs.AIcited by 1

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

Privacy deletion requests often arrive sequentially, creating a continual unlearning challenge for deployed multimodal large language models (MLLMs). However, existing benchmarks mainly focus on static or short-sequence settings, offering limited support for evaluating continual privacy deletion on privacy-critical documents. To bridge this gap, we introduce ICU-Bench, an Identity-centric Continual Unlearning benchmark (pronounced ``I see you'') for privacy-critical multimodal documents. ICU-Bench contains 1,000 synthetic privacy-sensitive profiles from medical reports and labor contracts, comprising 9,500 document images, 16,000 question-answer pairs, and 100 sequential forget tasks. We further introduce history-aware evaluation protocols and sequence-aware metrics to assess current forgetting, historical forgetting preservation, retained utility, and stability throughout the unlearning sequence. Experiments with representative unlearning methods on two MLLMs reveal that methods effective on current targets often fail to preserve earlier forgetting or retained capabilities over long sequences. Some methods further obtain low forget accuracy through severe model degradation. These results expose long-horizon failure modes overlooked by conventional evaluations and highlight the need for multimodal unlearning methods explicitly designed for continual privacy deletion. Our code and benchmark are publicly available at: https://github.com/AstorYH/ICU-Bench.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 34 references that are in the map. First posted on arXiv 7 May 2026.

Builds on

31 of 34 references.

Closest ancestors · 7

Same direction, newest first

  1. 2025PULSE: Practical Evaluation Scenarios for Large Multimodal Model Unlearning2 Jul
  2. 2025PEBench: A Fictitious Dataset to Benchmark Machine Unlearning for Multimodal Large Language Models16 Mar
  3. 2025Modality-Aware Neuron Pruning for Unlearning in Multimodal Large Language Models21 Feb
  4. 2025MMUnlearner: Reformulating Multimodal Machine Unlearning in the Era of Multimodal Large Language Models16 Feb
  5. 2024Protecting Privacy in Multimodal Large Language Models with MLLMU-Bench29 Oct
  6. 2024Cross-Modal Safety Alignment: Is textual unlearning all you need?27 May
  7. 2024Single Image Unlearning: Efficient Machine Unlearning in Multimodal Large Language Models21 May

Other references · 24

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  4. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  5. 2024TOFU: A Task of Fictitious Unlearning for LLMscited by 507
  6. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearningcited by 484
  7. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Modelscited by 424
  8. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  9. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Modelscited by 344
  10. 2023Large Language Model Unlearningcited by 288
  11. 2021Unrolling SGD: Understanding Factors Influencing Machine Unlearningcited by 274
  12. 2024MUSE: Machine Unlearning Six-Way Evaluation for Language Modelscited by 251
12 more
  1. 2024Rethinking Machine Unlearning for Large Language Modelscited by 247
  2. 2022Continual Learning and Private Unlearningcited by 124
  3. 2020Variational Bayesian Unlearningcited by 114
  4. 2024On Large Language Model Continual Unlearningcited by 51
  5. 2023Knowledge Unlearning for LLMs: Tasks, Methods, and Challengescited by 37
  6. 2024CLEAR: Character Unlearning in Textual and Visual Modalitiescited by 36
  7. 2024WAGLE: Strategic Weight Attribution for Effective and Modular Unlearning in Large Language Modelscited by 25
  8. 2024EFUF: Efficient Fine-grained Unlearning Framework for Mitigating Hallucinations in Multimodal Large Language Modelscited by 16
  9. 2024MU-Bench: A Multitask Multimodal Benchmark for Machine Unlearningcited by 16
  10. 2025Yuan: Yielding Unblemished Aesthetics Through A Unified Network for Visual Imperfections Removal in Generated Imagescited by 13
  11. 2025VideoEraser: Concept Erasure in Text-to-Video Diffusion Modelscited by 12
  12. 2025ForgetMe: Evaluating Selective Forgetting in Generative Modelscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2026Toward Fine-Grained Forgetting:Attribute Unlearning for Multimodal Large Language Models
  2. 2025SafeEraser: Enhancing Safety in Multimodal Large Language Models through Multimodal Machine Unlearning
  3. 2025Towards Benign Memory Forgetting for Selective Multimodal Large Language Model Unlearning
  4. 2026Operationalising the Right to be Forgotten in LLMs: A Lightweight Sequential Unlearning Framework for Privacy-Aligned Deployment in Politically Sensitive Environments
  5. 2026Open-MMUnlearning: Unifying Methods and Evaluation for MLLM Unlearning
  6. 2026PPE-Bench: A Benchmark for Evaluating MLLM Unlearning under Private-Public Entanglement
  7. 2026Beyond Visual Evidence: Revealing and Mitigating Relational Privacy Leakage in Document MLLMs
  8. 2025From Learning to Unlearning: Biomedical Security Protection in Multimodal Large Language Models