mlsanityData to 7 Oct 2026

Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Models

arXiv 2402.12048 ↗First posted 19 Feb 2024cs.CLcited by 54

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Catastrophic forgetting emerges as a critical challenge when fine-tuning multi-modal large language models (MLLMs), where improving performance on unseen tasks often leads to a significant performance drop on the original tasks. This paper presents a comprehensive analysis of catastrophic forgetting in MLLMs and introduces a post-training adjustment method called Model Tailor. Our method primarily preserves the pre-trained parameters while replacing a small number (≤ 10%) of fine-tuned parameters, maintaining ~ 99% effectiveness on original tasks versus pre-training, and achieving ~ 97% on new tasks compared to standard fine-tuning. Specifically, we derive a sparse mask to identify the "model patch", based on a fusion strategy that integrates salience and sensitivity analysis. Subsequently, a compensation mechanism is introduced to "decorate the patch", enhancing the model's performance on both target and original tasks. Additionally, our method is adaptable to multi-task scenarios. Through extensive experiments on InstructBLIP and LLaVA-1.5 in both image captioning and visual question answering tasks, our approach demonstrates significant task adaptability while preserving inherent pre-trained capabilities.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 54 of its 70 references that are in the map. First posted on arXiv 19 Feb 2024.

Builds on

54 of 70 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Improved Baselines with Visual Instruction Tuning5 Oct
  2. 2023Investigating the Catastrophic Forgetting in Multimodal Large Language Models19 Sep
  3. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul
  4. 2023A Survey on Multimodal Large Language Models23 Jun
  5. 2023MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models23 Jun
  6. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning11 May
  7. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuning5 May

Other references · 47

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2023Visual Instruction Tuningcited by 7,792
  8. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  9. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  10. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  11. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  12. 2015VQA: Visual Question Answeringcited by 2,489
35 more
  1. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  2. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  3. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  4. 2019Towards VQA Models That Can Readcited by 1,696
  5. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  6. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,461
  7. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  8. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  9. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  10. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  11. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  12. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  13. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 702
  14. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  15. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 600
  16. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  17. 2018Progress & Compress: A scalable framework for continual learningcited by 516
  18. 2020Class-incremental learning: survey and performance evaluation on image classificationcited by 446
  19. 2018nocaps: novel object captioning at scalecited by 444
  20. 2022Optimal Brain Compression: A Framework for Accurate Post-Training Quantization and Pruningcited by 317
  21. 2023InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Compositioncited by 297
  22. 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 254
  23. 2020Few-Shot Class-Incremental Learningcited by 249
  24. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  25. 2018Explicit Inductive Bias for Transfer Learning with Convolutional Networkscited by 226
  26. 2018Online Structured Laplace Approximations For Overcoming Catastrophic Forgettingcited by 197
  27. 2020Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgettingcited by 169
  28. 2019Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Modelscited by 141
  29. 2023Neural Collapse Inspired Feature-Classifier Alignment for Few-Shot Class Incremental Learningcited by 112
  30. 2023Task-Specific Skill Localization in Fine-tuned Language Modelscited by 104
  31. 2023Towards Language Models That Can See: Computer Vision Through the LENS of Natural Languagecited by 53
  32. 2021How Should Pre-Trained Language Models Be Fine-Tuned Towards Adversarial Robustness?cited by 49
  33. 2023Continual Instruction Tuning for Large Multimodal Modelscited by 49
  34. 2021Controlling Conditional Language Models without Catastrophic Forgettingcited by 32
  35. 2023ECoFLaP: Efficient Coarse-to-Fine Layer-Wise Pruning for Vision-Language Modelscited by 19

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 16; 2025: 27; 2026: 11.

54 papers.

Sort:

By area:

  1. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  2. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  3. 2025Lifelong Learning of Large Language Model based Agents: A Roadmapcited by 78
  4. 2024CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuningcited by 41
  5. 2025SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuningcited by 32
  6. 2024ModalPrompt: Towards Efficient Multimodal Continual Instruction Tuning with Dual-Modality Guided Promptcited by 29
  7. 2025Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuningcited by 27
  8. 2024DiffClass: Diffusion-Based Class Incremental Learningcited by 23
  9. 2024Unlocking Continual Learning Abilities in Language Modelscited by 22
  10. 2024Continual LLaVA: Continual Instruction Tuning in Large Vision-Language Modelscited by 22
  11. 2024Separate the Wheat from the Chaff: A Post-Hoc Approach to Safety Re-Alignment for Fine-Tuned Language Modelscited by 19
  12. 2025Vision-Language Modeling Meets Remote Sensing: Models, Datasets and Perspectivescited by 18
38 more
  1. 2024SAFE: Slow and Fast Parameter-Efficient Tuning for Continual Learning with Pre-Trained Modelscited by 17
  2. 2025Safe Delta: Consistently Preserving Safety when Fine-Tuning LLMs on Diverse Datasetscited by 17
  3. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Modelcited by 15
  4. 2024Mitigating the Backdoor Effect for Multi-Task Model Merging via Safety-Aware Subspacecited by 14
  5. 2024MoFO: Momentum-Filtered Optimizer for Mitigating Forgetting in LLM Fine-Tuningcited by 13
  6. 2025Mitigating Visual Knowledge Forgetting in MLLM Instruction-tuning via Modality-decoupled Gradient Descentcited by 13
  7. 2025LoRASculpt: Sculpting LoRA for Harmonizing General and Specialized Knowledge in Multimodal Large Language Modelscited by 11
  8. 2025X-Fusion: Introducing New Modality to Frozen Large Language Modelscited by 11
  9. 2024FusionBench: A Unified Library and Comprehensive Benchmark for Deep Model Fusioncited by 10
  10. 2025How to Teach Large Multimodal Models New Skillscited by 10
  11. 2024KIF: Knowledge Identification and Fusion for Language Model Continual Learningcited by 9
  12. 2025ZeroFlow: Overcoming Catastrophic Forgetting is Easier than You Thinkcited by 9
  13. 2025Scalable Model Merging with Progressive Layer-wise Distillationcited by 9
  14. 2025Merge before Forget: A Single LoRA Continual Learning via Continual Mergingcited by 9
  15. 2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 8
  16. 2025Knowledge Fusion of Large Language Models Via Modular SkillPackscited by 8
  17. 2025Locate-then-Merge: Neuron-Level Parameter Fusion for Mitigating Catastrophic Forgetting in Multimodal LLMscited by 7
  18. 2024Towards Incremental Learning in Large Language Models: A Critical Reviewcited by 6
  19. 2025Towards Benign Memory Forgetting for Selective Multimodal Large Language Model Unlearningcited by 6
  20. 2025Mitigating Intra- and Inter-modal Forgetting in Continual Learning of Unified Multimodal Modelscited by 6
  21. 2024BadCLM: Backdoor Attack in Clinical Language Models for Electronic Health Recordscited by 5
  22. 2026When Shared Knowledge Hurts: Spectral Over-Accumulation in Model Mergingcited by 5
  23. 2025Dynamic Context-oriented Decomposition for Task-aware Low-rank Adaptation with Less Forgetting and Faster Convergencecited by 4
  24. 2025A Systematic Study of In-the-Wild Model Merging for Large Language Modelscited by 4
  25. 2025Scalable Strategies for Continual Learning with Replaycited by 3
  26. 2025Merging Smarter, Generalizing Better: Enhancing Model Merging on OOD Datacited by 3
  27. 2025CLIP-SVD: Efficient and Interpretable Vision-Language Adaptation via Singular Valuescited by 3
  28. 2024MOMA: Masked Orthogonal Matrix Alignment for Zero-Additional-Parameter Model Mergingcited by 2
  29. 2025Memorization and Knowledge Injection in Gated LLMscited by 2
  30. 2025EvoEdit: Lifelong Free-Text Knowledge Editing through Latent Perturbation Augmentation and Knowledge-driven Parameter Fusioncited by 2
  31. 2025Mitigating Forgetting in Low Rank Adaptationcited by 2
  32. 2025MAGIC: Achieving Superior Model Merging via Magnitude Calibrationcited by 2
  33. 2026Model-Dowser: Data-Free Importance Probing to Mitigate Catastrophic Forgetting in Multimodal Large Language Modelscited by 2
  34. 2026LinguDistill: Recovering Linguistic Ability in Vision-Language Models via Selective Cross-Modal Distillationcited by 1
  35. 2026DiMtextsuperscript3: Bridging Multilingual and Multimodal Models via Direction- and Magnitude-Aware Mergingcited by 1
  36. 2026Dynamic Model Merging Made Slimcited by 1
  37. 2026Hidden Forgetting in Continual Multimodal Learning: When Accuracy Survives but Grounding Failscited by 1
  38. 2026MLUBench: A Benchmark for Lifelong Unlearning Evaluation in MLLMscited by 0

50 most cited of 54 All 54 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Fine-tuning MLLMs Without Forgetting Is Easier Than You Think
  2. 2024Learn from Downstream and Be Yourself in Multimodal Large Language Model Fine-Tuning
  3. 2026AWARe: Mitigating Catastrophic Forgetting via Activation-Weighted Adaptive REtention
  4. 2024Modality-Inconsistent Continual Learning of Multimodal Large Language Models
  5. 2024Examining Forgetting in Continual Pre-training of Aligned Large Language Models
  6. 2024HFT: Half Fine-Tuning for Large Language Models
  7. 2024Skipping Computations in Multimodal LLMs
  8. 2026Talking to Yourself: Defying Forgetting in Large Language Models