mlsanityData to 7 Oct 2026

Open-MMUnlearning: Unifying Methods and Evaluation for MLLM Unlearning

arXiv 2610.10358 ↗First posted 7 Oct 2026cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness also fits: MLLM training and visual instruction tuning

also in 1 more
Abstract (arXiv)

As multimodal large language models (MLLMs) become more capable and widely deployed, concerns about privacy and safety have become increasingly pressing. Machine unlearning offers one approach to addressing these concerns by removing designated information from trained models while preserving unrelated capabilities. However, fragmented implementations and evaluation protocols, incomplete robustness testing, and limited understanding of metric reliability make progress in MLLM unlearning difficult to assess systematically. We introduce Open-MMUnlearning, an open-source, extensible framework that integrates target-model preparation, multimodal data processing, unlearning, and evaluation through shared interfaces and structured configurations. The framework supports five benchmarks spanning privacy, safety, and copyright, eight MLLMs from four model families, and twelve unlearning methods. Its evaluation suite jointly assesses forgetting effectiveness, retained utility, and robustness to model interventions, adversarial inputs, and membership inference attacks. Using a common evaluation protocol, we compare ten representative unlearning methods. In this comparison, GD and MIP-Editor tie for the highest overall score: GD achieves the highest Forget Quality, while MIP-Editor preserves more Model Utility. We further introduce a metric meta-evaluation protocol that tests faithfulness using models with controlled exposure to target knowledge and robustness under quantization and relearning. Among the thirteen evaluated metrics, BLEU achieves the highest aggregate reliability score. KS-Test attains the highest faithfulness AUC but performs less well on robustness. Together, the framework and these findings support reproducible comparison of MLLM unlearning methods and systematic assessment of evaluation reliability.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 60 references that are in the map. First posted on arXiv 7 Oct 2026.

Builds on

26 of 60 references.

Closest ancestors · 14

Same direction, newest first

  1. 2026Does Forgetting Transfer Across Modalities? A Real-World Benchmark for Cross-Modal Knowledge Unlearning Evaluation4 Aug
  2. 2026Visual-Noise Guided In-Context Distillation for Multimodal Large Language Model Unlearning26 May
  3. 2026On the Robustness of Machine Unlearning for Vision-Language Models26 May
  4. 2026PPU-Bench:Real World Benchmark for Personalized Partial Unlearning in Vision Language Models9 May
  5. 2026Erase Persona, Forget Lore: Benchmarking Multimodal Copyright Unlearning in Large Vision Language Models5 May
  6. 2026Which Concepts to Forget and How to Refuse? Decomposing Concepts for Continual Unlearning in Large Vision-Language Models23 Mar
  7. 2026Knowledge Vector Weakening: Efficient Training-free Unlearning for Large Vision-Language Models29 Jan
  8. 2025Towards Benign Memory Forgetting for Selective Multimodal Large Language Model Unlearning25 Nov
  9. 2025SineProject: Machine Unlearning for Stable Vision Language Alignment23 Nov
  10. 2025OFFSIDE: Benchmarking Unlearning Misinformation in Multimodal Large Language Models26 Oct
  11. 2025MLLMEraser: Achieving Test-Time Unlearning in Multimodal Large Language Models through Activation Steering5 Oct
  12. 2025From Learning to Unlearning: Biomedical Security Protection in Multimodal Large Language Models6 Aug
2 more
  1. 2025Unlearning Sensitive Information in Multimodal LLMs: Benchmark and Attack-Defense Evaluation1 May
  2. 2025SAUCE: Selective Concept Unlearning in Vision-Language Models with Sparse Autoencoders16 Mar

Other references · 12

  1. 2025Qwen2.5-VL Technical Reportcited by 5,418
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  3. 2025Gemma 3 Technical Reportcited by 1,686
  4. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 488
  5. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearningcited by 484
  6. 2024Eight Methods to Evaluate Robust Unlearning in LLMscited by 153
  7. 2025Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyondcited by 76
  8. 2024Visual-RolePlay: Universal Jailbreak Attack on MultiModal Large Language Models via Role-playing Image Charactercited by 61
  9. 2024From Theft to Bomb-Making: The Ripple Effect of Unlearning in Defending Against Jailbreak Attackscited by 29
  10. 2024Pandora's White-Box: Precise Training Data Detection and Extraction in Large Language Modelscited by 11
  11. 2026Object Hallucination-Free Reinforcement Unlearning for Vision-Language Modelscited by 3
  12. 2025Robust MLLM Unlearning via Visual Knowledge Distillationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025SafeEraser: Enhancing Safety in Multimodal Large Language Models through Multimodal Machine Unlearning
  2. 2026MLUBench: A Benchmark for Lifelong Unlearning Evaluation in MLLMs
  3. 2024Protecting Privacy in Multimodal Large Language Models with MLLMU-Bench
  4. 2026Exploring and Bridging Knowledge Holes in Unlearned Multimodal Large Language Models
  5. 2023Large Language Model Unlearning
  6. 2026A Model Merging Approach for Continual MLLM Unlearning
  7. 2025MMUnlearner: Reformulating Multimodal Machine Unlearning in the Era of Multimodal Large Language Models
  8. 2024Rethinking Machine Unlearning for Large Language Models