mlsanityData to 7 Oct 2026

Metric Unreliability in Multimodal Machine Unlearning: A Systematic Analysis and Principled Unified Score

arXiv 2605.02206 ↗First posted 4 May 2026cs.CV cs.LGcited by 0

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

Machine unlearning in Vision-Language Models (VLMs) is required for compliance with the General Data Protection Regulation (GDPR), yet current evaluation practices are inconsistent. We present the first systematic study of metric reliability in multimodal unlearning. Five standard metrics, Forget Accuracy (FA), Retain Accuracy (RA), Membership Inference Attack (MIA), Activation Distance (AD), and JS divergence (JS), yield conflicting method rankings across three VQA benchmarks (MLLMU-Bench, UnLOK-VQA, MMUBench). Kendall tau analysis over 36 unlearned LLaVA-1.5-7B models reveals two opposing clusters, {FA, RA, MIA} and {AD, JS}, with tau_FA_AD = -0.26, reproduced on BLIP-2 OPT-2.7B. Agreement is lower in multimodal VQA (average tau = 0.086) than in unimodal classification (average tau = 0.158; difference = 0.072), indicating that dual image-and-text pathways amplify inconsistency. We introduce the Unified Quality Score (UQS), a composite metric with weights derived from each metric's Spearman correlation with the oracle distance d(M_hat, M_star), where M_star is the oracle model retrained only on the retain set. RA shows the strongest reliability (rho = 0.484, p = 0.003), while FA is negatively correlated (rho = -0.418, p = 0.011). UQS yields stable rankings under 100 random weight perturbations (tau = 0.647 +- 0.262). We release the benchmark, 36 checkpoints, and an interactive leaderboard. Code and pre-computed results are available at https://github.com/neurips26/UnifiedUnl.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 34 of its 45 references that are in the map. First posted on arXiv 4 May 2026.

Builds on

34 of 45 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Unlearning Sensitive Information in Multimodal LLMs: Benchmark and Attack-Defense Evaluation1 May
  2. 2024Protecting Privacy in Multimodal Large Language Models with MLLMU-Bench29 Oct
  3. 2024Single Image Unlearning: Efficient Machine Unlearning in Multimodal Large Language Models21 May

Other references · 31

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Visual Instruction Tuningcited by 7,792
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  9. 2015VQA: Visual Question Answeringcited by 2,489
  10. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  11. 2019Machine Unlearningcited by 1,020
  12. 2021Membership Inference Attacks From First Principlescited by 839
19 more
  1. 2019Eternal Sunshine of the Spotless Net: Selective Forgetting in Deep Networkscited by 599
  2. 2024TOFU: A Task of Fictitious Unlearning for LLMscited by 507
  3. 2019Making AI Forget You: Data Deletion in Machine Learningcited by 447
  4. 2022Memory-Based Model Editing at Scalecited by 443
  5. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Modelscited by 424
  6. 2023SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generationcited by 376
  7. 2023Detecting Pretraining Data from Large Language Modelscited by 375
  8. 2021Remember What You Want to Forget: Algorithms for Machine Unlearningcited by 360
  9. 2022A Survey of Machine Unlearningcited by 298
  10. 2023Large Language Model Unlearningcited by 288
  11. 2024MUSE: Machine Unlearning Six-Way Evaluation for Language Modelscited by 251
  12. 2024Rethinking Machine Unlearning for Large Language Modelscited by 247
  13. 2022Can Bad Teaching Induce Forgetting? Unlearning in Deep Networks using an Incompetent Teachercited by 189
  14. 2023Fast Machine Unlearning Without Retraining Through Selective Synaptic Dampeningcited by 177
  15. 2020Dataset Security for Machine Learning: Data Poisoning, Backdoor Attacks, and Defensescited by 171
  16. 2020Mixed-Privacy Forgetting in Deep Networkscited by 158
  17. 2020DeltaGrad: Rapid retraining of machine learning modelscited by 143
  18. 2023Boundary Unlearningcited by 132
  19. 2025OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metricscited by 80

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026On the Robustness of Machine Unlearning for Vision-Language Models
  2. 2026Open-MMUnlearning: Unifying Methods and Evaluation for MLLM Unlearning
  3. 2026Multimodal Unlearning Across Vision, Language, Video, and Audio: Survey of Methods, Datasets, and Benchmarks
  4. 2025PEBench: A Fictitious Dataset to Benchmark Machine Unlearning for Multimodal Large Language Models
  5. 2024Benchmarking Vision Language Model Unlearning via Fictitious Facial Identity Dataset
  6. 2026Visual-Noise Guided In-Context Distillation for Multimodal Large Language Model Unlearning
  7. 2024CLEAR: Character Unlearning in Textual and Visual Modalities
  8. 2026Stochastic Meta-Unlearning: Bridging Language Backbone and Multimodal Unlearning