mlsanityData to 14 Oct 2026

Benchmarking Vision Language Model Unlearning via Fictitious Facial Identity Dataset

arXiv 2411.03554 ↗First posted 5 Nov 2024cs.CVcited by 30

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

Machine unlearning has emerged as an effective strategy for forgetting specific information in the training data. However, with the increasing integration of visual data, privacy concerns in Vision Language Models (VLMs) remain underexplored. To address this, we introduce Facial Identity Unlearning Benchmark (FIUBench), a novel VLM unlearning benchmark designed to robustly evaluate the effectiveness of unlearning algorithms under the Right to be Forgotten setting. Specifically, we formulate the VLM unlearning task via constructing the Fictitious Facial Identity VQA dataset and apply a two-stage evaluation pipeline that is designed to precisely control the sources of information and their exposure levels. In terms of evaluation, since VLM supports various forms of ways to ask questions with the same semantic meaning, we also provide robust evaluation metrics including membership inference attacks and carefully designed adversarial privacy attacks to evaluate the performance of algorithms. Through the evaluation of four baseline VLM unlearning algorithms within FIUBench, we find that all methods remain limited in their unlearning performance, with significant trade-offs between model utility and forget quality. Furthermore, our findings also highlight the importance of privacy attacks for robust evaluations. We hope FIUBench will drive progress in developing more effective VLM unlearning algorithms.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 17 of its 66 references that are in the map. First posted on arXiv 5 Nov 2024.

Builds on

17 of 66 references.

References · 17

  1. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  2. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Modelscited by 1,652
  3. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  4. 2024TOFU: A Task of Fictitious Unlearning for LLMscited by 533
  5. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 509
  6. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearningcited by 504
  7. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 422
  8. 2023Detecting Pretraining Data from Large Language Modelscited by 387
  9. 2022A Survey of Machine Unlearningcited by 312
  10. 2023Large Language Model Unlearningcited by 302
  11. 2024Machine Unlearning of Pre-trained Large Language Modelscited by 119
  12. 2024SOUL: Unlocking the Power of Second-Order Optimization for LLM Unlearningcited by 104
5 more
  1. 2023Right to be Forgotten in the Era of Large Language Models: Implications, Challenges, and Solutionscited by 60
  2. 2023Exploring the Landscape of Machine Unlearning: A Comprehensive Survey and Taxonomycited by 51
  3. 2023A Challenger to GPT-4V? Early Explorations of Gemini in Visual Expertisecited by 49
  4. 2024MIKE: A New Benchmark for Fine-grained Multimodal Entity Knowledge Editingcited by 25
  5. 2023MultiDelete for Multimodal Machine Unlearningcited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 13; 2026: 17.

30 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  2. 2025A Comprehensive Survey of Machine Unlearning Techniques for Large Language Modelscited by 34
  3. 2025SafeEraser: Enhancing Safety in Multimodal Large Language Models through Multimodal Machine Unlearningcited by 32
  4. 2025ReLearn: Unlearning via Learning for Large Language Modelscited by 22
  5. 2025Learning to Unlearn while Retaining: Combating Gradient Conflicts in Machine Unlearningcited by 19
  6. 2025Safety Mirage: How Spurious Correlations Undermine VLM Safety Fine-Tuning and Can Be Mitigated by Machine Unlearningcited by 16
  7. 2025SAUCE: Selective Concept Unlearning in Vision-Language Models with Sparse Autoencoderscited by 13
  8. 2025R-TOFU: Unlearning in Large Reasoning Modelscited by 9
  9. 2025Existing Large Language Model Unlearning Evaluations Are Inconclusivecited by 8
  10. 2026Multimodal Unlearning Across Vision, Language, Video, and Audio: Survey of Methods, Datasets, and Benchmarkscited by 7
  11. 2025Uncovering the Potential Risks in Unlearning: Danger of English-only Unlearning in Multilingual LLMscited by 4
  12. 2026Knowledge Vector Weakening: Efficient Training-free Unlearning for Large Vision-Language Modelscited by 3
18 more
  1. 2026SALMUBench: A Benchmark for Sensitive Association-Level Multimodal Unlearningcited by 3
  2. 2026Visual-Noise Guided In-Context Distillation for Multimodal Large Language Model Unlearningcited by 3
  3. 2026Erase Persona, Forget Lore: Benchmarking Multimodal Copyright Unlearning in Large Vision Language Modelscited by 2
  4. 2026PreUnlearn: Auditing Collateral Knowledge Damage Before Large Language Model Unlearningcited by 2
  5. 2025Cross-Modal Attention Guided Unlearning in Vision-Language Modelscited by 1
  6. 2026Suppression Is Not Forgetting: Residual Recoverability in Visual Concept Unlearning for VLMscited by 1
  7. 2026PPU-Bench:Real World Benchmark for Personalized Partial Unlearning in Vision Language Modelscited by 1
  8. 2026MLUBench: A Benchmark for Lifelong Unlearning Evaluation in MLLMscited by 1
  9. 2026Does Forgetting Transfer Across Modalities? A Real-World Benchmark for Cross-Modal Knowledge Unlearning Evaluationcited by 1
  10. 2026Beyond Cross-Lingual Transfer: Benchmarking Propagation Boundaries in Multilingual LLM Unlearningcited by 1
  11. 2025Rethinking Post-Unlearning Behavior of Large Vision-Language Modelscited by 0
  12. 2025Lifting Data-Tracing Machine Unlearning to Knowledge-Tracing for Foundation Modelscited by 0
  13. 2026RankLLM: Weighted Ranking of LLMs by Quantifying Question Difficultycited by 0
  14. 2026Stochastic Meta-Unlearning: Bridging Language Backbone and Multimodal Unlearningcited by 0
  15. 2026Unlearning Under Imbalance: Benchmarking Fairness in Multimodal LLM Unlearningcited by 0
  16. 2026Toward Fine-Grained Forgetting:Attribute Unlearning for Multimodal Large Language Modelscited by 0
  17. 2026Exploring and Bridging Knowledge Holes in Unlearned Multimodal Large Language Modelscited by 0
  18. 2026Are We Really Measuring Forgetting? Pitfalls in Evaluating Machine Unlearning for Multimodal LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Towards Machine Unlearning Benchmarks: Forgetting the Personal Identities in Facial Recognition Systems
  2. 2025PEBench: A Fictitious Dataset to Benchmark Machine Unlearning for Multimodal Large Language Models
  3. 2026On the Robustness of Machine Unlearning for Vision-Language Models
  4. 2024Gone but Not Forgotten: Improved Benchmarks for Machine Unlearning
  5. 2026PPE-Bench: A Benchmark for Evaluating MLLM Unlearning under Private-Public Entanglement
  6. 2026Metric Unreliability in Multimodal Machine Unlearning: A Systematic Analysis and Principled Unified Score
  7. 2024The Phantom Menace: Unmasking Privacy Leakages in Vision-Language Models
  8. 2026Before Forgetting, Learn to Remember: Revisiting Foundational Learning Failures in LVLM Unlearning Benchmarks