mlsanityData to 7 Oct 2026

Model-Dowser: Data-Free Importance Probing to Mitigate Catastrophic Forgetting in Multimodal Large Language Models

arXiv 2602.04509 ↗First posted 4 Feb 2026cs.CLcited by 2

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Fine-tuning Multimodal Large Language Models (MLLMs) on task-specific data is an effective way to improve performance on downstream applications. However, such adaptation often leads to a degradation in generalization on pretrained tasks, a phenomenon known as Catastrophic Forgetting. Existing methods that aim to mitigate this issue either become ineffective when fine-tuning deeper layers of the language decoder or scale poorly with increasing model size. To address these limitations, we propose Model-Dowser, a novel sparse fine-tuning approach for MLLMs. Model-Dowser measures a principled importance score for each model parameter with respect to pretrained generalization (prior to downstream adaptation) by jointly considering weight magnitudes, input activations, and output sensitivities. During fine-tuning, Model-Dowser selectively preserves high-importance parameters and updates the remaining. Comprehensive experiments on two representative MLLMs, LLaVA and NVILA, demonstrate that Model-Dowser effectively mitigates catastrophic forgetting and consistently outperforms prior methods, while remaining resource-efficient and scalable to multi-billion-parameter models.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 54 of its 70 references that are in the map. First posted on arXiv 4 Feb 2026.

Builds on

54 of 70 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025MLLM-CL: Continual Learning for Multimodal Large Language Models5 Jun
  2. 2025SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuning5 May
  3. 2025Federated Continual Instruction Tuning17 Mar
  4. 2024CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Model13 Mar
  5. 2023Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Models12 Mar
  6. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling6 Nov

Other references · 48

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  5. 2023Segment Anythingcited by 8,158
  6. 2023Visual Instruction Tuningcited by 7,792
  7. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  9. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  12. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
36 more
  1. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,048
  2. 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
  3. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  4. 2020GLU Variants Improve Transformercited by 1,724
  5. 2019Towards VQA Models That Can Readcited by 1,696
  6. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,573
  7. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  8. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  9. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  10. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
  11. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  12. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  13. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  14. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  15. 2024An Image is Worth 1/2 Tokens After Layer 2: Plug-and-Play Inference Acceleration for Large Vision-Language Modelscited by 629
  16. 2020Pruning neural networks without any data by iteratively conserving synaptic flowcited by 475
  17. 2020Class-incremental learning: survey and performance evaluation on image classificationcited by 446
  18. 2024Massive Activations in Large Language Modelscited by 277
  19. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 262
  20. 2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 228
  21. 2018Explicit Inductive Bias for Transfer Learning with Convolutional Networkscited by 226
  22. 2024NVILA: Efficient Frontier Visual Language Modelscited by 206
  23. 2025LLaVA-Mini: Efficient Image and Video Large Multimodal Models with One Vision Tokencited by 164
  24. 2023Language Models Meet World Models: Embodied Experiences Enhance Language Modelscited by 123
  25. 2023Neural Collapse Inspired Feature-Classifier Alignment for Few-Shot Class Incremental Learningcited by 112
  26. 2024Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Modelscited by 112
  27. 2023Task-Specific Skill Localization in Fine-tuned Language Modelscited by 104
  28. 2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 66
  29. 2024Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Modelscited by 54
  30. 2021How Should Pre-Trained Language Models Be Fine-Tuned Towards Adversarial Robustness?cited by 49
  31. 2023Gradient-based Parameter Selection for Efficient Fine-Tuningcited by 40
  32. 2022Parameter-Efficient Sparsity for Large Language Models Fine-Tuningcited by 27
  33. 2023Overcoming Generic Knowledge Loss with Selective Parameter Updatecited by 23
  34. 2024An Empirical Study on Parameter-Efficient Fine-Tuning for MultiModal Large Language Modelscited by 22
  35. 2024Random Masking Finds Winning Tickets for Parameter Efficient Fine-tuningcited by 16
  36. 2024SPP: Sparsity-Preserved Parameter-Efficient Fine-Tuning for Large Language Modelscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026From Structure to Synergy: A Survey of Vision-Language Perception Paradigm Evolution in Multimodal Large Language Modelscited by 0
  2. 2026What Should We Freeze? Guarded Freezing: Connectivity Shapes the Fine-Tuning of Pretrained Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026AWARe: Mitigating Catastrophic Forgetting via Activation-Weighted Adaptive REtention
  2. 2024Learn from Downstream and Be Yourself in Multimodal Large Language Model Fine-Tuning
  3. 2026Fine-tuning MLLMs Without Forgetting Is Easier Than You Think
  4. 2024Examining Forgetting in Continual Pre-training of Aligned Large Language Models
  5. 2024Balancing Speciality and Versatility: A Coarse to Fine Framework for Mitigating Catastrophic Forgetting in Large Language Models
  6. 2026Mechanistic Analysis of Catastrophic Forgetting in Large Language Models During Continual Fine-tuning
  7. 2023Investigating the Catastrophic Forgetting in Multimodal Large Language Models
  8. 2025Towards Benign Memory Forgetting for Selective Multimodal Large Language Model Unlearning