mlsanityData to 7 Oct 2026

Modality-Aware Neuron Pruning for Unlearning in Multimodal Large Language Models

arXiv 2502.15910 ↗First posted 21 Feb 2025cs.CLcited by 25

On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness

also in 1 more
Abstract (arXiv)

Generative models such as Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs) trained on massive datasets can lead them to memorize and inadvertently reveal sensitive information, raising ethical and privacy concerns. While some prior works have explored this issue in the context of LLMs, it presents a unique challenge for MLLMs due to the entangled nature of knowledge across modalities, making comprehensive unlearning more difficult. To address this challenge, we propose Modality Aware Neuron Unlearning (MANU), a novel unlearning framework for MLLMs designed to selectively clip neurons based on their relative importance to the targeted forget data, curated for different modalities. Specifically, MANU consists of two stages: important neuron selection and selective pruning. The first stage identifies and collects the most influential neurons across modalities relative to the targeted forget knowledge, while the second stage is dedicated to pruning those selected neurons. MANU effectively isolates and removes the neurons that contribute most to the forget data within each modality, while preserving the integrity of retained knowledge. Our experiments conducted across various MLLM architectures illustrate that MANU can achieve a more balanced and comprehensive unlearning in each modality without largely affecting the overall model utility.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 49 of its 64 references that are in the map. First posted on arXiv 21 Feb 2025.

Builds on

49 of 64 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Protecting Privacy in Multimodal Large Language Models with MLLMU-Bench29 Oct

Other references · 48

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  10. 2020Extracting Training Data from Large Language Modelscited by 2,144
  11. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  12. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
36 more
  1. 2023mPLUG-Owl: Modularization Empowers Large Language Models with Multimodalitycited by 1,038
  2. 2019Machine Unlearningcited by 1,020
  3. 2022Mass-Editing Memory in a Transformercited by 976
  4. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  5. 2019Are Sixteen Heads Really Better than One?cited by 803
  6. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  7. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  8. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  9. 2024TOFU: A Task of Fictitious Unlearning for LLMscited by 507
  10. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 488
  11. 2023Is ChatGPT a General-Purpose Natural Language Processing Task Solver?cited by 486
  12. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearningcited by 484
  13. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  14. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 404
  15. 2021Unrolling SGD: Understanding Factors Influencing Machine Unlearningcited by 274
  16. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 149
  17. 2024Towards Safer Large Language Models through Machine Unlearningcited by 134
  18. 2022Continual Learning and Private Unlearningcited by 124
  19. 2020Variational Bayesian Unlearningcited by 114
  20. 2024An Adversarial Perspective on Machine Unlearning for AI Safetycited by 113
  21. 2024Arc2Face: A Foundation Model for ID-Consistent Human Facescited by 101
  22. 2022What do Vision Transformers Learn? A Visual Explorationcited by 75
  23. 2024DE-COP: Detecting Copyrighted Content in Language Models Training Datacited by 72
  24. 2024Personalized Pieces: Efficient Personalized Large Language Models through Collaborative Effortscited by 71
  25. 2024MIA-Bench: Towards Better Instruction Following Evaluation of Multimodal LLMscited by 62
  26. 2024Machine Unlearning in Generative AI: A Surveycited by 49
  27. 2024Dissecting Language Models: Machine Unlearning via Selective Pruningcited by 38
  28. 2024SHIELD: Evaluation and Defense Strategies for Copyright Compliance in LLM Text Generationcited by 33
  29. 2025Towards Trustworthy Retrieval Augmented Generation for Large Language Models: A Surveycited by 29
  30. 2017Large-scale Cloze Test Dataset Created by Teacherscited by 27
  31. 2024Towards Robust Evaluation of Unlearning in LLMs via Data Transformationscited by 19
  32. 2024Can LLMs Convert Graphs to Text-Attributed Graphs?cited by 19
  33. 2024MOPI-HFRS: A Multi-objective Personalized Health-aware Food Recommendation System with LLM-enhanced Interpretationcited by 18
  34. 2024Commonsense Knowledge Editing Based on Free-Text in LLMscited by 16
  35. 2024AmoebaLLM: Constructing Any-Shape Large Language Models for Efficient and Instant Deploymentcited by 12
  36. 2025Pretrained Image-Text Models are Secretly Video Captionerscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 9; 2026: 16.

25 papers.

Sort:

By area:

  1. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  2. 2025Model Unlearning via Sparse Autoencoder Subspace Guided Projectionscited by 17
  3. 2025SoK: Machine Unlearning for Large Language Modelscited by 9
  4. 2025MLLMEraser: Achieving Test-Time Unlearning in Multimodal Large Language Models through Activation Steeringcited by 9
  5. 2025Towards Benign Memory Forgetting for Selective Multimodal Large Language Model Unlearningcited by 6
  6. 2026Missing-by-Design: Certifiable Modality Deletion for Revocable Multimodal Sentiment Analysiscited by 6
  7. 2025Overcoming Multi-step Complexity in Multimodal Theory-of-Mind Reasoning: A Scalable Bayesian Plannercited by 5
  8. 2025Generative Model Unlearning: A Survey through Target Events, Unlearning Operators, and Evaluation Protocolscited by 4
  9. 2026Knowledge Vector Weakening: Efficient Training-free Unlearning for Large Vision-Language Modelscited by 3
  10. 2026Visual-Guided Key-Token Regularization for Multimodal Large Language Model Unlearningcited by 3
  11. 2026Null Space Constrained Contrastive Visual Forgetting for MLLM Unlearningcited by 3
  12. 2026Object Hallucination-Free Reinforcement Unlearning for Vision-Language Modelscited by 3
13 more
  1. 2025Towards Reasoning-Preserving Unlearning in Multimodal Large Language Modelscited by 2
  2. 2026ActErase: A Training-Free Paradigm for Precise Concept Erasure via Activation Redirectioncited by 2
  3. 2025Robust MLLM Unlearning via Visual Knowledge Distillationcited by 1
  4. 2026ICU-Bench:Benchmarking Continual Unlearning in Multimodal Large Language Modelscited by 1
  5. 2026PPU-Bench:Real World Benchmark for Personalized Partial Unlearning in Vision Language Modelscited by 1
  6. 2026Relationship-Aware Safety Unlearning for Multimodal LLMscited by 0
  7. 2026SPACE: Source-free Proxy Anchor Concept Erasure for MLLMscited by 0
  8. 2026Output Vector Editing for Memorization Mitigation in Large Language Modelscited by 0
  9. 2026Detect, Unlearn, Restore: Defending Text Summarization Models Against Data Poisoningcited by 0
  10. 2026POPS: Recovering Unlearned Multi-Modality Knowledge in MLLMs with Prompt-Optimized Parameter Shakingcited by 0
  11. 2026When Structured Sparse Autoencoders Learn Consistent Concepts Across Modalitiescited by 0
  12. 2026One Modality to Forget Them All: Enhancing Cross-Modal Unlearning in Vision-Language Modelscited by 0
  13. 2026Stochastic Meta-Unlearning: Bridging Language Backbone and Multimodal Unlearningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Cross-Modal Unlearning via Influential Neuron Path Editing in Multimodal Large Language Models
  2. 2025Sparse-Autoencoder-Guided Internal Representation Unlearning for Large Language Models
  3. 2024MINER: Mining the Underlying Pattern of Modality-Specific Neurons in Multimodal Large Language Models
  4. 2026Visual-Noise Guided In-Context Distillation for Multimodal Large Language Model Unlearning
  5. 2025MMUnlearner: Reformulating Multimodal Machine Unlearning in the Era of Multimodal Large Language Models
  6. 2026PRMU: A Corpus-Free Benchmark for Person-Centric Knowledge Unlearning in Multimodal Large Language Models
  7. 2026Exploring and Bridging Knowledge Holes in Unlearned Multimodal Large Language Models
  8. 2025PEBench: A Fictitious Dataset to Benchmark Machine Unlearning for Multimodal Large Language Models