mlsanityData to 7 Oct 2026

Single Image Unlearning: Efficient Machine Unlearning in Multimodal Large Language Models

arXiv 2405.12523 ↗First posted 21 May 2024cs.CV cs.AIcited by 44

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

Machine unlearning empowers individuals with the `right to be forgotten' by removing their private or sensitive information encoded in machine learning models. However, it remains uncertain whether MU can be effectively applied to Multimodal Large Language Models (MLLMs), particularly in scenarios of forgetting the leaked visual data of concepts. To overcome the challenge, we propose an efficient method, Single Image Unlearning (SIU), to unlearn the visual recognition of a concept by fine-tuning a single associated image for few steps. SIU consists of two key aspects: (i) Constructing Multifaceted fine-tuning data. We introduce four targets, based on which we construct fine-tuning data for the concepts to be forgotten; (ii) Jointly training loss. To synchronously forget the visual recognition of concepts and preserve the utility of MLLMs, we fine-tune MLLMs through a novel Dual Masked KL-divergence Loss combined with Cross Entropy loss. Alongside our method, we establish MMUBench, a new benchmark for MU in MLLMs and introduce a collection of metrics for its evaluation. Experimental results on MMUBench show that SIU completely surpasses the performance of existing methods. Furthermore, we surprisingly find that SIU can avoid invasive membership inference attacks and jailbreak attacks. To the best of our knowledge, we are the first to explore MU in MLLMs. We will release the code and benchmark in the near future.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 47 of its 53 references that are in the map. First posted on arXiv 21 May 2024.

Builds on

47 of 53 references.

References · 47

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  8. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  10. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  11. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  12. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
35 more
  1. 2019Towards VQA Models That Can Readcited by 1,696
  2. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  3. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
  4. 2023PaLM 2 Technical Reportcited by 1,154
  5. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  6. 2023A Survey on Multimodal Large Language Modelscited by 898
  7. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  8. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  9. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  10. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 548
  11. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  12. 2024TOFU: A Task of Fictitious Unlearning for LLMscited by 507
  13. 2023SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generationcited by 376
  14. 2023Detecting Pretraining Data from Large Language Modelscited by 375
  15. 2023Who's Harry Potter? Approximate Unlearning in LLMscited by 366
  16. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreakercited by 337
  17. 2023MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questionscited by 292
  18. 2023Large Language Model Unlearningcited by 288
  19. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  20. 2024Rethinking Machine Unlearning for Large Language Modelscited by 247
  21. 2023Multilingual Jailbreak Challenges in Large Language Modelscited by 222
  22. 2023Unlearn What You Want to Forget: Efficient Unlearning for LLMscited by 211
  23. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  24. 2024Machine Unlearning of Pre-trained Large Language Modelscited by 118
  25. 2023Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Modelscited by 112
  26. 2023KGA: A General Machine Unlearning Framework Based on Knowledge Gap Alignmentcited by 94
  27. 2023Fast Model Debias with Machine Unlearningcited by 70
  28. 2023Learning to Unlearn: Instance-wise Unlearning for Pre-trained Classifierscited by 66
  29. 2024Can AI Assistants Know What They Don't Know?cited by 47
  30. 2023Knowledge Unlearning for LLMs: Tasks, Methods, and Challengescited by 37
  31. 2021Improved Fine-Tuning by Better Leveraging Pre-Training Datacited by 36
  32. 2023Never Train from Scratch: Fair Comparison of Long-Sequence Models Requires Data-Driven Priorscited by 33
  33. 2023On the Connection between Pre-training Data Diversity and Fine-tuning Robustnesscited by 25
  34. 2024MIKE: A New Benchmark for Fine-grained Multimodal Entity Knowledge Editingcited by 25
  35. 2023Learning and Forgetting Unsafe Examples in Large Language Modelscited by 24

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 21; 2026: 18.

44 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  2. 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 81
  3. 2025A Survey of Safety on Large Vision-Language Models: Attacks, Defenses and Evaluationscited by 66
  4. 2024A Survey on Evaluation of Multimodal Large Language Modelscited by 46
  5. 2025Unlearning Isn't Deletion: Investigating Reversibility of Machine Unlearning in LLMscited by 41
  6. 2025A Comprehensive Survey of Machine Unlearning Techniques for Large Language Modelscited by 30
  7. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  8. 2024CLIPErase: Efficient Unlearning of Visual-Textual Associations in CLIPcited by 19
  9. 2024MU-Bench: A Multitask Multimodal Benchmark for Machine Unlearningcited by 16
  10. 2025SAUCE: Selective Concept Unlearning in Vision-Language Models with Sparse Autoencoderscited by 12
  11. 2025Unlearning Sensitive Information in Multimodal LLMs: Benchmark and Attack-Defense Evaluationcited by 11
  12. 2025Image Can Bring Your Memory Back: A Novel Multi-Modal Guided Attack against Image Generation Model Unlearningcited by 9
32 more
  1. 2025MLLMEraser: Achieving Test-Time Unlearning in Multimodal Large Language Models through Activation Steeringcited by 9
  2. 2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 8
  3. 2025OFFSIDE: Benchmarking Unlearning Misinformation in Multimodal Large Language Modelscited by 7
  4. 2025PEBench: A Fictitious Dataset to Benchmark Machine Unlearning for Multimodal Large Language Modelscited by 6
  5. 2025Towards Benign Memory Forgetting for Selective Multimodal Large Language Model Unlearningcited by 6
  6. 2026Missing-by-Design: Certifiable Modality Deletion for Revocable Multimodal Sentiment Analysiscited by 6
  7. 2025PULSE: Practical Evaluation Scenarios for Large Multimodal Model Unlearningcited by 5
  8. 2025Fairness and Robustness in Machine Unlearningcited by 4
  9. 2025Forget-MI: Machine Unlearning for Forgetting Multimodal Information in Healthcare Settingscited by 3
  10. 2025SineProject: Machine Unlearning for Stable Vision Language Alignmentcited by 3
  11. 2026Visual-Guided Key-Token Regularization for Multimodal Large Language Model Unlearningcited by 3
  12. 2026Null Space Constrained Contrastive Visual Forgetting for MLLM Unlearningcited by 3
  13. 2026Object Hallucination-Free Reinforcement Unlearning for Vision-Language Modelscited by 3
  14. 2026Visual-Noise Guided In-Context Distillation for Multimodal Large Language Model Unlearningcited by 3
  15. 2025SNS-Bench-VL: Benchmarking Multimodal Large Language Models in Social Networking Servicescited by 2
  16. 2026CATA: Continual Machine Unlearning via Conflict-Averse Task Arithmeticcited by 2
  17. 2025GUARD: Guided Unlearning and Retention via Data Attribution for Large Language Modelscited by 1
  18. 2025SUGAR: A Sweeter Spot for Generative Unlearning of Many Identitiescited by 1
  19. 2025Robust MLLM Unlearning via Visual Knowledge Distillationcited by 1
  20. 2026Which Concepts to Forget and How to Refuse? Decomposing Concepts for Continual Unlearning in Large Vision-Language Modelscited by 1
  21. 2026Suppression Is Not Forgetting: Residual Recoverability in Visual Concept Unlearning for VLMscited by 1
  22. 2026ICU-Bench:Benchmarking Continual Unlearning in Multimodal Large Language Modelscited by 1
  23. 2026PPU-Bench:Real World Benchmark for Personalized Partial Unlearning in Vision Language Modelscited by 1
  24. 2025Lifting Data-Tracing Machine Unlearning to Knowledge-Tracing for Foundation Modelscited by 0
  25. 2026Metric Unreliability in Multimodal Machine Unlearning: A Systematic Analysis and Principled Unified Scorecited by 0
  26. 2026Classification-Head Bias in Class-Level Machine Unlearning: Diagnosis, Mitigation, and Evaluationcited by 0
  27. 2026SPACE: Source-free Proxy Anchor Concept Erasure for MLLMscited by 0
  28. 2026MLUBench: A Benchmark for Lifelong Unlearning Evaluation in MLLMscited by 0
  29. 2026POPS: Recovering Unlearned Multi-Modality Knowledge in MLLMs with Prompt-Optimized Parameter Shakingcited by 0
  30. 2026Stochastic Meta-Unlearning: Bridging Language Backbone and Multimodal Unlearningcited by 0
  31. 2026Unlearning Under Imbalance: Benchmarking Fairness in Multimodal LLM Unlearningcited by 0
  32. 2026Partition-Aware Unlearning for Removing Spurious Correlations in Large Vision-Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026ICED: Concept-level Machine Unlearning via Interpretable Concept Decomposition
  2. 2025AUVIC: Adversarial Unlearning of Visual Concepts for Multi-modal Large Language Models
  3. 2025MMUnlearner: Reformulating Multimodal Machine Unlearning in the Era of Multimodal Large Language Models
  4. 2026Toward Fine-Grained Forgetting:Attribute Unlearning for Multimodal Large Language Models
  5. 2025SafeEraser: Enhancing Safety in Multimodal Large Language Models through Multimodal Machine Unlearning
  6. 2026Knowledge Vector Weakening: Efficient Training-free Unlearning for Large Vision-Language Models
  7. 2024CLEAR: Character Unlearning in Textual and Visual Modalities
  8. 2026AIM: Anchor Identity Features, Then Match for Multimodal Large Language Model Unlearning