mlsanityData to 14 Oct 2026

Improving Multimodal Large Language Models Using Continual Learning

arXiv 2410.19925 ↗First posted 25 Oct 2024cs.CL cs.CV cs.LGcited by 5

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Generative large language models (LLMs) exhibit impressive capabilities, which can be further augmented by integrating a pre-trained vision model into the original LLM to create a multimodal LLM (MLLM). However, this integration often significantly decreases performance on natural language understanding and generation tasks, compared to the original LLM. This study investigates this issue using the LLaVA MLLM, treating the integration as a continual learning problem. We evaluate five continual learning methods to mitigate forgetting and identify a technique that enhances visual understanding while minimizing linguistic performance loss. Our approach reduces linguistic performance degradation by up to 15% over the LLaVA recipe, while maintaining high multimodal accuracy. We also demonstrate the robustness of our method through continual learning on a sequence of vision-language tasks, effectively preserving linguistic skills while acquiring new multimodal capabilities. Project webpage: https://shikhar-srivastava.github.io/cl-for-improving-mllms

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 31 of its 37 references that are in the map. First posted on arXiv 25 Oct 2024.

Builds on

31 of 37 references.

References · 31

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  10. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  11. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  12. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,333
19 more
  1. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  2. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  3. 2019Towards VQA Models That Can Readcited by 1,718
  4. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  5. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  6. 2023VILA: On Pre-training for Visual Language Modelscited by 692
  7. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 668
  8. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  9. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 639
  10. 2023Textbooks Are All You Needcited by 569
  11. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 565
  12. 2023MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Actioncited by 517
  13. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 339
  14. 2023A Rank Stabilization Scaling Factor for Fine-Tuning with LoRAcited by 243
  15. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 232
  16. 2022Linearly Mapping from Image to Text Spacecited by 142
  17. 2023Grounding Language Models to Images for Multimodal Inputs and Outputscited by 131
  18. 2023Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Modelscited by 101
  19. 2023Overcoming the Stability Gap in Continual Learningcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 2.

5 papers.

  1. 2024How to Merge Your Multimodal Models Over Time?cited by 23
  2. 2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 9
  3. 2025Efficient Few-Shot Continual Learning in Vision-Language Modelscited by 2
  4. 2026LinguDistill: Recovering Linguistic Ability in Vision-Language Models via Selective Cross-Modal Distillationcited by 1
  5. 2026CheXmix: Unified Generative Pretraining for Vision Language Models in Medical Imagingcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025MLLM-CL: Continual Learning for Multimodal Large Language Models
  2. 2024Continual Learning for Large Language Models: A Survey
  3. 2025Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting
  4. 2024Modality-Inconsistent Continual Learning of Multimodal Large Language Models
  5. 2023Continual Instruction Tuning for Large Multimodal Models
  6. 2026Is Our Benchmark Enough? An Analysis of Continual Learning for MLLMs
  7. 2025Merge then Realign: Simple and Effective Modality-Incremental Continual Learning for Multimodal LLMs
  8. 2026Fine-tuning MLLMs Without Forgetting Is Easier Than You Think