mlsanityData to 7 Oct 2026

Visual-Guided Key-Token Regularization for Multimodal Large Language Model Unlearning

arXiv 2601.22020 ↗First posted 29 Jan 2026cs.LG cs.CVcited by 3

On the map: Multimodal LLMs / vision-language models → Visual token pruning and efficiency

also in 1 more
Abstract (arXiv)

Unlearning in Multimodal Large Language Models (MLLMs) prevents the model from revealing private information when queried about target images. Existing MLLM unlearning methods largely adopt approaches developed for LLMs. They treat all answer tokens uniformly, disregarding their varying importance in the unlearning process. Moreover, these methods focus exclusively on the language modality, disregarding visual cues that indicate key tokens in answers. In this paper, after formulating the problem of unlearning in multimodal question answering for MLLMs, we propose Visual-Guided Key-Token Regularization (ViKeR). We leverage irrelevant visual inputs to predict ideal post-unlearning token-level distributions and use these distributions to regularize the unlearning process, thereby prioritizing key tokens. Further, we define key tokens in unlearning via information entropy and discuss ViKeR's effectiveness through token-level gradient reweighting, which amplifies updates on key tokens. Experiments on MLLMU and CLEAR benchmarks demonstrate that our method effectively performs unlearning while mitigating forgetting and maintaining response coherence.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Visual token pruning and efficiency, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 36 of its 45 references that are in the map. First posted on arXiv 29 Jan 2026.

Builds on

36 of 45 references.

References · 36

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  4. 2019Machine Unlearningcited by 1,020
  5. 2023A Survey on Multimodal Large Language Modelscited by 898
  6. 2024TOFU: A Task of Fictitious Unlearning for LLMscited by 507
  7. 2024The WMDP Benchmark: Measuring and Reducing Malicious Use With Unlearningcited by 488
  8. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearningcited by 484
  9. 2019Making AI Forget You: Data Deletion in Machine Learningcited by 447
  10. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Modelscited by 424
  11. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  12. 2023Large Language Model Unlearningcited by 288
24 more
  1. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
  2. 2024MUSE: Machine Unlearning Six-Way Evaluation for Language Modelscited by 251
  3. 2023In-Context Unlearning: Language Models as Few Shot Unlearnerscited by 214
  4. 2023Multimodal Large Language Models: A Surveycited by 193
  5. 2024MLLM-Protector: Ensuring MLLM's Safety without Hurting Performancecited by 111
  6. 2023Copyright Violations and Large Language Modelscited by 108
  7. 2024SOUL: Unlocking the Power of Second-Order Optimization for LLM Unlearningcited by 104
  8. 2025OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metricscited by 80
  9. 2025Towards LLM Unlearning Resilient to Relearning Attacks: A Sharpness-Aware Minimization Perspective and Beyondcited by 76
  10. 2024Protecting Privacy in Multimodal Large Language Models with MLLMU-Benchcited by 61
  11. 2025Rethinking LLM Unlearning Objectives: A Gradient Perspective and Go Beyondcited by 59
  12. 2024Single Image Unlearning: Efficient Machine Unlearning in Multimodal Large Language Modelscited by 44
  13. 2025Exploring Criteria of Loss Reweighting to Enhance LLM Unlearningcited by 39
  14. 2024CLEAR: Character Unlearning in Textual and Visual Modalitiescited by 36
  15. 2025SafeEraser: Enhancing Safety in Multimodal Large Language Models through Multimodal Machine Unlearningcited by 31
  16. 2025MMUnlearner: Reformulating Multimodal Machine Unlearning in the Era of Multimodal Large Language Modelscited by 29
  17. 2024Towards Effective Evaluations and Comparisons for LLM Unlearning Methodscited by 27
  18. 2025Modality-Aware Neuron Pruning for Unlearning in Multimodal Large Language Modelscited by 25
  19. 2024Cross-Modal Safety Alignment: Is textual unlearning all you need?cited by 22
  20. 2024Split, Unlearn, Merge: Leveraging Data Attributes for More Effective Unlearning in LLMscited by 18
  21. 2024EFUF: Efficient Fine-grained Unlearning Framework for Mitigating Hallucinations in Multimodal Large Language Modelscited by 16
  22. 2025GRU: Mitigating the Trade-off between Unlearning and Retention for LLMscited by 11
  23. 2025Cross-Modal Unlearning via Influential Neuron Path Editing in Multimodal Large Language Modelscited by 7
  24. 2025AUVIC: Adversarial Unlearning of Visual Concepts for Multi-modal Large Language Modelscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026On the Robustness of Machine Unlearning for Vision-Language Modelscited by 1
  2. 2026AIM: Anchor Identity Features, Then Match for Multimodal Large Language Model Unlearningcited by 1
  3. 2026Where Identity Lives: Localized, Retain-Free Identity Unlearning in Multimodal Large Language Modelscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Cross-Modal Attention Guided Unlearning in Vision-Language Models
  2. 2024freePruner: A Training-free Approach for Large Multimodal Model Acceleration
  3. 2024Efficient Large Multi-modal Models via Visual Context Compression
  4. 2024PAR: Prompt-Aware Token Reduction Method for Efficient Large Multimodal Models
  5. 2026Visual-Noise Guided In-Context Distillation for Multimodal Large Language Model Unlearning
  6. 2024Recoverable Compression: A Multimodal Vision Token Recovery Mechanism Guided by Text Information
  7. 2025MLLMEraser: Achieving Test-Time Unlearning in Multimodal Large Language Models through Activation Steering
  8. 2026Attend, Transform, or Silence: Operator-Level Visual Skipping for Efficient Multimodal LLM Inference