mlsanityData to 14 Oct 2026

SafeEraser: Enhancing Safety in Multimodal Large Language Models through Multimodal Machine Unlearning

arXiv 2502.12520 ↗First posted 18 Feb 2025cs.CVcited by 32

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

As Multimodal Large Language Models (MLLMs) develop, their potential security issues have become increasingly prominent. Machine Unlearning (MU), as an effective strategy for forgetting specific knowledge in training data, has been widely used in privacy protection. However, MU for safety in MLLM has yet to be fully explored. To address this issue, we propose SAFEERASER, a safety unlearning benchmark for MLLMs, consisting of 3,000 images and 28.8K VQA pairs. We comprehensively evaluate unlearning methods from two perspectives: forget quality and model utility. Our findings show that existing MU methods struggle to maintain model performance while implementing the forget operation and often suffer from over-forgetting. Hence, we introduce Prompt Decouple (PD) Loss to alleviate over-forgetting through decouple prompt during unlearning process. To quantitatively measure over-forgetting mitigated by PD Loss, we propose a new metric called Safe Answer Refusal Rate (SARR). Experimental results demonstrate that combining PD Loss with existing unlearning methods can effectively prevent over-forgetting and achieve a decrease of 79.5% in the SARR metric of LLaVA-7B and LLaVA-13B, while maintaining forget quality and model utility. Our code and dataset will be released upon acceptance. Warning: This paper contains examples of harmful language and images, and reader discretion is recommended.

Key facts. Cited by 32 papers in the map (data through 14 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 40 of its 53 references that are in the map. First posted on arXiv 18 Feb 2025.

Builds on

40 of 53 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024VLSBench: Unveiling Visual Leakage in Multimodal Safety29 Nov
  2. 2024Benchmarking Vision Language Model Unlearning via Fictitious Facial Identity Dataset5 Nov
  3. 2024Protecting Privacy in Multimodal Large Language Models with MLLMU-Bench29 Oct
  4. 2024JailBreakV: A Benchmark for Assessing the Robustness of MultiModal Large Language Models against Jailbreak Attacks3 Apr
  5. 2024Safety of Multimodal Large Language Models on Images and Texts1 Feb
  6. 2023FigStep: Jailbreaking Large Vision-Language Models via Typographic Visual Prompts9 Nov
  7. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Models26 Jul

Other references · 33

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  5. 2023Qwen Technical Reportcited by 3,344
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  7. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  8. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  9. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,239
  10. 2019Machine Unlearningcited by 1,061
  11. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  12. 2024TOFU: A Task of Fictitious Unlearning for LLMscited by 533
21 more
  1. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrationscited by 431
  2. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 422
  3. 2023Who's Harry Potter? Approximate Unlearning in LLMscited by 381
  4. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreakercited by 367
  5. 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 278
  6. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easilycited by 259
  7. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 230
  8. 2023In-Context Unlearning: Language Models as Few Shot Unlearnerscited by 224
  9. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lenscited by 143
  10. 2024Towards Safer Large Language Models through Machine Unlearningcited by 139
  11. 2024An Adversarial Perspective on Machine Unlearning for AI Safetycited by 123
  12. 2024Large Language Model Unlearning via Embedding-Corrupted Promptscited by 116
  13. 2023KGA: A General Machine Unlearning Framework Based on Knowledge Gap Alignmentcited by 99
  14. 2023UrbanCLIP: Learning Text-enhanced Urban Region Profiling with Contrastive Language-Image Pretraining from the Webcited by 78
  15. 2024A Survey of Mathematical Reasoning in the Era of Multimodal Large Language Model: Benchmark, Method & Challengescited by 55
  16. 2025Open Problems in Machine Unlearning for AI Safetycited by 51
  17. 2024Mitigating Modality Prior-Induced Hallucinations in Multimodal Large Language Models via Deciphering Attention Causalitycited by 42
  18. 2024ErrorRadar: Benchmarking Complex Mathematical Reasoning of Multimodal Large Language Models Via Error Detectioncited by 37
  19. 2024Assessment of Multimodal Large Language Models in Alignment with Human Valuescited by 36
  20. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  21. 2024Does Unlearning Truly Unlearn? A Black Box Evaluation of LLM Unlearning Methodscited by 26

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 17; 2026: 15.

32 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  2. 2025LLM Agents for Education: Advances and Applicationscited by 59
  3. 2025AlphaSteer: Learning Refusal Steering with Principled Null-Space Constraintcited by 49
  4. 2025MathAgent: Leveraging a Mixture-of-Math-Agent Framework for Real-World Multimodal Mathematical Error Detectioncited by 24
  5. 2025DocPruner: A Storage-Efficient Framework for Multi-Vector Visual Document Retrieval via Adaptive Patch-Level Embedding Pruningcited by 20
  6. 2025Model Unlearning via Sparse Autoencoder Subspace Guided Projectionscited by 18
  7. 2025GUARD: Generation-time LLM Unlearning via Adaptive Restriction and Detectioncited by 17
  8. 2025Safety Mirage: How Spurious Correlations Undermine VLM Safety Fine-Tuning and Can Be Mitigated by Machine Unlearningcited by 16
  9. 2025VLA-Mark: A cross modal watermark for large vision-language alignment modelcited by 12
  10. 2025MLLMEraser: Achieving Test-Time Unlearning in Multimodal Large Language Models through Activation Steeringcited by 10
  11. 2025OFFSIDE: Benchmarking Unlearning Misinformation in Multimodal Large Language Modelscited by 8
  12. 2025Cross-Modal Unlearning via Influential Neuron Path Editing in Multimodal Large Language Modelscited by 8
20 more
  1. 2026BalDRO: A Distributionally Robust Optimization based Framework for Large Language Model Unlearningcited by 8
  2. 2025CAFES: A Collaborative Multi-Agent Framework for Multi-Granular Multimodal Essay Scoringcited by 7
  3. 2025DRAGON: Guard LLM Unlearning in Context via Negative Detection and Reasoningcited by 7
  4. 2026Multimodal Unlearning Across Vision, Language, Video, and Audio: Survey of Methods, Datasets, and Benchmarkscited by 7
  5. 2025LM-mixup: Text Data Augmentation via Language Model based Mixupcited by 3
  6. 2025SineProject: Machine Unlearning for Stable Vision Language Alignmentcited by 3
  7. 2026A Visual Semantic Adaptive Watermark grounded by Prefix-Tuning for Large Vision-Language Modelcited by 3
  8. 2026Visual-Guided Key-Token Regularization for Multimodal Large Language Model Unlearningcited by 3
  9. 2026CAMEO: A Conditional and Quality-Aware Multi-Agent Image Editing Orchestratorcited by 3
  10. 2026Visual-Noise Guided In-Context Distillation for Multimodal Large Language Model Unlearningcited by 3
  11. 2026Position: The Term "Machine Unlearning" Is Overused in LLMscited by 3
  12. 2025Towards Mitigating Excessive Forgetting in LLM Unlearning via Entanglement-Guidance with Proxy Constraintcited by 1
  13. 2026Which Concepts to Forget and How to Refuse? Decomposing Concepts for Continual Unlearning in Large Vision-Language Modelscited by 1
  14. 2026ASRU: Activation Steering Meets Reinforcement Unlearning for Multimodal Large Language Modelscited by 1
  15. 2026One Modality to Forget Them All: Enhancing Cross-Modal Unlearning in Vision-Language Modelscited by 0
  16. 2026Stochastic Meta-Unlearning: Bridging Language Backbone and Multimodal Unlearningcited by 0
  17. 2026Unlearning Under Imbalance: Benchmarking Fairness in Multimodal LLM Unlearningcited by 0
  18. 2026Partition-Aware Unlearning for Removing Spurious Correlations in Large Vision-Language Modelscited by 0
  19. 2026AUV-Bench: Aesthetic Understanding and Generation Evaluation for User Interfacescited by 0
  20. 2026CUSE: Fine-Grained Multimodal Unlearning with Counterfactual Guidance and On-Policy Self-Distillationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Open-MMUnlearning: Unifying Methods and Evaluation for MLLM Unlearning
  2. 2026Exploring and Bridging Knowledge Holes in Unlearned Multimodal Large Language Models
  3. 2025Towards Benign Memory Forgetting for Selective Multimodal Large Language Model Unlearning
  4. 2026Unlearning in LLMs: Methods, Evaluation, and Open Challenges
  5. 2026Forgetting Only What Matters: Layer-Selective Unlearning toward Robust LLMs
  6. 2025BLUR: A Benchmark for LLM Unlearning Robust to Forget-Retain Overlap
  7. 2024A Closer Look at Machine Unlearning for Large Language Models
  8. 2024Soft Prompting for Unlearning in Large Language Models