mlsanityData to 13 Oct 2026

Protecting Privacy in Multimodal Large Language Models with MLLMU-Bench

arXiv 2410.22108 ↗First posted 29 Oct 2024cs.CL cs.AIcited by 62

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

Generative models such as Large Language Models (LLM) and Multimodal Large Language models (MLLMs) trained on massive web corpora can memorize and disclose individuals' confidential and private data, raising legal and ethical concerns. While many previous works have addressed this issue in LLM via machine unlearning, it remains largely unexplored for MLLMs. To tackle this challenge, we introduce Multimodal Large Language Model Unlearning Benchmark (MLLMU-Bench), a novel benchmark aimed at advancing the understanding of multimodal machine unlearning. MLLMU-Bench consists of 500 fictitious profiles and 153 profiles for public celebrities, each profile feature over 14 customized question-answer pairs, evaluated from both multimodal (image+text) and unimodal (text) perspectives. The benchmark is divided into four sets to assess unlearning algorithms in terms of efficacy, generalizability, and model utility. Finally, we provide baseline results using existing generative model unlearning algorithms. Surprisingly, our experiments show that unimodal unlearning algorithms excel in generation and cloze tasks, while multimodal unlearning approaches perform better in classification tasks with multimodal inputs.

Key facts. Cited by 62 papers in the map (data through 13 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 40 of its 58 references that are in the map. First posted on arXiv 29 Oct 2024.

Builds on

40 of 58 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Cross-Modal Safety Alignment: Is textual unlearning all you need?27 May
  2. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models26 Jul

Other references · 38

  1. 2020Language Models are Few-Shot Learnerscited by 32,320
  2. 2022Training language models to follow instructions with human feedbackcited by 15,856
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,492
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,986
  6. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,000
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,317
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,389
  9. 2014Deep Learning Face Attributes in the Wildcited by 3,686
  10. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,810
  11. 2016Deep Learning with Differential Privacycited by 2,718
  12. 2020Extracting Training Data from Large Language Modelscited by 2,181
26 more
  1. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,057
  2. 2019Machine Unlearningcited by 1,056
  3. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,040
  4. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 708
  5. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 659
  6. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 546
  7. 2024TOFU: A Task of Fictitious Unlearning for LLMscited by 519
  8. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 498
  9. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearningcited by 496
  10. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 488
  11. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 427
  12. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 407
  13. 2021Unrolling SGD: Understanding Factors Influencing Machine Unlearningcited by 277
  14. 2023SneakyPrompt: Jailbreaking Text-to-image Generative Modelscited by 187
  15. 2024Towards Safer Large Language Models through Machine Unlearningcited by 136
  16. 2022Continual Learning and Private Unlearningcited by 129
  17. 2020Variational Bayesian Unlearningcited by 116
  18. 2024Arc2Face: A Foundation Model for ID-Consistent Human Facescited by 101
  19. 2024Democratizing Large Language Models via Personalized Parameter-Efficient Fine-tuningcited by 89
  20. 2024DE-COP: Detecting Copyrighted Content in Language Models Training Datacited by 73
  21. 2024Demystifying Verbatim Memorization in Large Language Modelscited by 56
  22. 2024Machine Unlearning in Generative AI: A Surveycited by 49
  23. 2024UnUnlearning: Unlearning is not sufficient for content regulation in advanced generative AIcited by 48
  24. 2024SHIELD: Evaluation and Defense Strategies for Copyright Compliance in LLM Text Generationcited by 34
  25. 2017Large-scale Cloze Test Dataset Created by Teacherscited by 27
  26. 2023Breaking the Trilemma of Privacy, Utility, Efficiency via Controllable Machine Unlearningcited by 22

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 27; 2026: 35.

62 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey of Machine Unlearning Techniques for Large Language Modelscited by 32
  2. 2025SafeEraser: Enhancing Safety in Multimodal Large Language Models through Multimodal Machine Unlearningcited by 31
  3. 2025MMUnlearner: Reformulating Multimodal Machine Unlearning in the Era of Multimodal Large Language Modelscited by 29
  4. 2025Temporal Working Memory: Query-Guided Segment Refinement for Enhanced Multimodal Understandingcited by 28
  5. 2025Modality-Aware Neuron Pruning for Unlearning in Multimodal Large Language Modelscited by 25
  6. 2025Safety Mirage: How Spurious Correlations Undermine VLM Safety Fine-Tuning and Can Be Mitigated by Machine Unlearningcited by 16
  7. 2025Position: Privacy Is Not Just Memorization!cited by 16
  8. 2025SAUCE: Selective Concept Unlearning in Vision-Language Models with Sparse Autoencoderscited by 12
  9. 2025Attention! Your Vision Language Model Could Be Maliciously Manipulatedcited by 12
  10. 2025MLLMEraser: Achieving Test-Time Unlearning in Multimodal Large Language Models through Activation Steeringcited by 9
  11. 2025VortexPIA: Indirect Prompt Injection Attack against LLMs for Efficient Extraction of User Privacycited by 7
  12. 2025OFFSIDE: Benchmarking Unlearning Misinformation in Multimodal Large Language Modelscited by 7
38 more
  1. 2025Cross-Modal Unlearning via Influential Neuron Path Editing in Multimodal Large Language Modelscited by 7
  2. 2025Steering Multimodal Large Language Models Decoding for Context-Aware Safetycited by 6
  3. 2025Towards Benign Memory Forgetting for Selective Multimodal Large Language Model Unlearningcited by 6
  4. 2026Missing-by-Design: Certifiable Modality Deletion for Revocable Multimodal Sentiment Analysiscited by 6
  5. 2026Multimodal Unlearning Across Vision, Language, Video, and Audio: Survey of Methods, Datasets, and Benchmarkscited by 6
  6. 2025Beyond Text: Unveiling Privacy Vulnerabilities in Multi-modal Retrieval-Augmented Generationcited by 5
  7. 2025Overcoming Multi-step Complexity in Multimodal Theory-of-Mind Reasoning: A Scalable Bayesian Plannercited by 5
  8. 2025PULSE: Practical Evaluation Scenarios for Large Multimodal Model Unlearningcited by 5
  9. 2025DUSK: Do Not Unlearn Shared Knowledgecited by 4
  10. 2025SUA: Stealthy Multimodal Large Language Model Unlearning Attackcited by 4
  11. 2025Generative Model Unlearning: A Survey through Target Events, Unlearning Operators, and Evaluation Protocolscited by 4
  12. 2025Uncovering the Potential Risks in Unlearning: Danger of English-only Unlearning in Multilingual LLMscited by 4
  13. 2025Watch Out Your Album! On the Inadvertent Privacy Memorization in Multi-Modal Large Language Modelscited by 3
  14. 2025From Learning to Unlearning: Biomedical Security Protection in Multimodal Large Language Modelscited by 3
  15. 2025SineProject: Machine Unlearning for Stable Vision Language Alignmentcited by 3
  16. 2026Knowledge Vector Weakening: Efficient Training-free Unlearning for Large Vision-Language Modelscited by 3
  17. 2026Visual-Guided Key-Token Regularization for Multimodal Large Language Model Unlearningcited by 3
  18. 2026SALMUBench: A Benchmark for Sensitive Association-Level Multimodal Unlearningcited by 3
  19. 2026Before Forgetting, Learn to Remember: Revisiting Foundational Learning Failures in LVLM Unlearning Benchmarkscited by 3
  20. 2026Null Space Constrained Contrastive Visual Forgetting for MLLM Unlearningcited by 3
  21. 2026Knowledge Beyond Language: Bridging the Gap in Multilingual Machine Unlearning Evaluationcited by 3
  22. 2026Visual-Noise Guided In-Context Distillation for Multimodal Large Language Model Unlearningcited by 3
  23. 2026Rethinking Visual Privacy: A Compositional Privacy Risk Framework for Severity Assessment with VLMscited by 2
  24. 2026Scaling Participation in Modular AI Systemscited by 2
  25. 2025Understanding Machine Unlearning Through the Lens of Mode Connectivitycited by 1
  26. 2025Cross-Modal Attention Guided Unlearning in Vision-Language Modelscited by 1
  27. 2026HearSay Benchmark: Do Audio LLMs Leak What They Hear?cited by 1
  28. 2026Suppression Is Not Forgetting: Residual Recoverability in Visual Concept Unlearning for VLMscited by 1
  29. 2026Metric Unreliability in Multimodal Machine Unlearning: A Systematic Analysis and Principled Unified Scorecited by 1
  30. 2026ICU-Bench:Benchmarking Continual Unlearning in Multimodal Large Language Modelscited by 1
  31. 2026PPU-Bench:Real World Benchmark for Personalized Partial Unlearning in Vision Language Modelscited by 1
  32. 2026Vision Language Model Helps Private Information De-Identification in Vision Datacited by 1
  33. 2026Does Forgetting Transfer Across Modalities? A Real-World Benchmark for Cross-Modal Knowledge Unlearning Evaluationcited by 1
  34. 2026AIM: Anchor Identity Features, Then Match for Multimodal Large Language Model Unlearningcited by 1
  35. 2026Where Identity Lives: Localized, Retain-Free Identity Unlearning in Multimodal Large Language Modelscited by 1
  36. 2026Membership Inference for Contrastive Pre-training Models with Text-only PII Queriescited by 0
  37. 2026Seeing Without Exposing: Adaptive Privacy Control for Open-World, Context-Hungry MLLMscited by 0
  38. 2026SPACE: Source-free Proxy Anchor Concept Erasure for MLLMscited by 0

50 most cited of 62 All 62 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Open-MMUnlearning: Unifying Methods and Evaluation for MLLM Unlearning
  2. 2026Unlearning in LLMs: Methods, Evaluation, and Open Challenges
  3. 2024Machine Unlearning in Large Language Models
  4. 2025PEBench: A Fictitious Dataset to Benchmark Machine Unlearning for Multimodal Large Language Models
  5. 2024MUSE: Machine Unlearning Six-Way Evaluation for Language Models
  6. 2024A Closer Look at Machine Unlearning for Large Language Models
  7. 2024Second-Order Information Matters: Revisiting Machine Unlearning for Large Language Models
  8. 2024The Frontier of Data Erasure: Machine Unlearning for Large Language Models