mlsanityData to 7 Oct 2026

CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Model

arXiv 2403.08350 ↗First posted 13 Mar 2024cs.CVcited by 60

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Instruction tuning represents a prevalent strategy employed by Multimodal Large Language Models (MLLMs) to align with human instructions and adapt to new tasks. Nevertheless, MLLMs encounter the challenge of adapting to users' evolving knowledge and demands. Therefore, how to retain existing skills while acquiring new knowledge needs to be investigated. In this paper, we present a comprehensive benchmark, namely Continual Instruction tuNing (CoIN), to assess existing MLLMs in the sequential instruction tuning paradigm. CoIN comprises 10 commonly used datasets spanning 8 task categories, ensuring a diverse range of instructions and tasks. Besides, the trained model is evaluated from two aspects: Instruction Following and General Knowledge, which assess the alignment with human intention and knowledge preserved for reasoning, respectively. Experiments on CoIN demonstrate that current powerful MLLMs still suffer catastrophic forgetting, and the failure in intention alignment assumes the main responsibility, instead of the knowledge forgetting. To this end, we introduce MoELoRA to MLLMs which is effective to retain the previous instruction alignment. Experimental results consistently illustrate the forgetting decreased from this method on CoIN.

Key facts. Cited by 60 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 57 of its 72 references that are in the map. First posted on arXiv 13 Mar 2024.

Builds on

57 of 72 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Continual Instruction Tuning for Large Multimodal Models27 Nov

Other references · 56

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2023Visual Instruction Tuningcited by 7,792
  9. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  11. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  12. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
38 more
  1. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  2. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  3. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  5. 2023A Survey of Large Language Modelscited by 2,537
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  7. 2016Learning without Forgettingcited by 2,465
  8. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  9. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  11. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  12. 2019Towards VQA Models That Can Readcited by 1,696
  13. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  14. 2016Progressive Neural Networkscited by 1,332
  15. 2017Memory Aware Synapses: Learning what (not) to forgetcited by 1,094
  16. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  17. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  18. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 895
  19. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 882
  20. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  21. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  22. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 672
  23. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  24. 2023MiniGPT-v2: large language model as a unified interface for vision-language multi-task learningcited by 617
  25. 2023Textbooks Are All You Needcited by 561
  26. 2018Progress & Compress: A scalable framework for continual learningcited by 516
  27. 2018Learning to Learn without Forgetting by Maximizing Transfer and Minimizing Interferencecited by 511
  28. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
  29. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  30. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  31. 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
  32. 2020Supermasks in Superpositioncited by 223
  33. 2021New Insights on Reducing Abrupt Representation Change in Online Continual Learningcited by 215
  34. 2021How Many Data Points is a Prompt Worth?cited by 197
  35. 2021VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioningcited by 190
  36. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70Bcited by 183
  37. 2023Investigating the Catastrophic Forgetting in Multimodal Large Language Modelscited by 119
  38. 2020Efficient Continual Learning with Modular Networks and Task-Driven Priorscited by 87

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 23; 2026: 29.

60 papers.

Sort:

By area:

  1. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  2. 2024A Survey on Benchmarks of Multimodal Large Language Modelscited by 81
  3. 2025Reinforcement Fine-Tuning Naturally Mitigates Forgetting in Continual Post-Trainingcited by 42
  4. 2025SEFE: Superficial and Essential Forgetting Eliminator for Multimodal Continual Instruction Tuningcited by 32
  5. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Modelscited by 27
  6. 2025Dynamic Mixture of Curriculum LoRA Experts for Continual Multimodal Instruction Tuningcited by 27
  7. 2024A Practitioner's Guide to Continual Multimodal Pretrainingcited by 22
  8. 2024Continual LLaVA: Continual Instruction Tuning in Large Vision-Language Modelscited by 22
  9. 2025MLLM-CL: Continual Learning for Multimodal Large Language Modelscited by 21
  10. 2024SMoLoRA: Exploring and Defying Dual Catastrophic Forgetting in Continual Visual Instruction Tuningcited by 19
  11. 2025Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgettingcited by 16
  12. 2025LLaVA-c: Continual Improved Visual Instruction Tuningcited by 14
38 more
  1. 2025Bisecle: Binding and Separation in Continual Learning for Video Language Understandingcited by 13
  2. 2025Federated Continual Instruction Tuningcited by 10
  3. 2025MCITlib: Multimodal Continual Instruction Tuning Library and Benchmarkcited by 10
  4. 2025How to Teach Large Multimodal Models New Skillscited by 10
  5. 2024Adapt-∞: Scalable Continual Multimodal Instruction Tuning via Dynamic Data Selectioncited by 9
  6. 2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 8
  7. 2026SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuningcited by 8
  8. 2025Continual Learning in Vision-Language Models via Aligned Model Mergingcited by 7
  9. 2025When Large Multimodal Models Confront Evolving Knowledge: Challenges and Explorationscited by 6
  10. 2025Multimodal Continual Instruction Tuning with Dynamic Gradient Guidancecited by 6
  11. 2025Mitigating Intra- and Inter-modal Forgetting in Continual Learning of Unified Multimodal Modelscited by 6
  12. 2025Task-Core Memory Management and Consolidation for Long-term Continual Learningcited by 4
  13. 2025Instruction-Grounded Visual Projectors for Continual Learning of Generative Vision-Language Modelscited by 4
  14. 2025Harmonious Parameter Adaptation in Continual Visual Instruction Tuning for Safety-Aligned MLLMscited by 3
  15. 2026Lifelong Embodied Navigation Learningcited by 3
  16. 2026Elastic Weight Consolidation Done Right for Continual Learningcited by 3
  17. 2025LLaVA-CMoE: Towards Continual Mixture of Experts for Large Vision-Language Modelscited by 2
  18. 2025MLLM-CTBench: A Benchmark for Continual Instruction Tuning with Reasoning Process Diagnosiscited by 2
  19. 2026Model-Dowser: Data-Free Importance Probing to Mitigate Catastrophic Forgetting in Multimodal Large Language Modelscited by 2
  20. 2026φ-DPO: Fairness Direct Preference Optimization Approach to Continual Learning in Large Multimodal Modelscited by 2
  21. 2026On Token's Dilemma: Dynamic MoE with Drift-Aware Token Assignment for Continual Learning of Large Vision Language Modelscited by 2
  22. 2026Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuningcited by 2
  23. 2026Janus-LoRA: A Balanced Low-Rank Adaptation for Continual Learningcited by 2
  24. 2025R^2MoE: Redundancy-Removal Mixture of Experts for Lifelong Concept Learningcited by 1
  25. 2026All-day Multi-scenes Lifelong Vision-and-Language Navigation with Tucker Adaptationcited by 1
  26. 2026CL-VISTA: Benchmarking Continual Learning in Video Large Language Modelscited by 1
  27. 2026DSCA: Dynamic Subspace Concept Alignment for Lifelong VLM Editingcited by 1
  28. 2026Attention-Spectrum Regularization for Replay-Free Continual Multimodal LLMscited by 1
  29. 2026Hidden Forgetting in Continual Multimodal Learning: When Accuracy Survives but Grounding Failscited by 1
  30. 2026CL-Anomaly: Layer-Adaptive Mixture-of-Experts with Multimodal Large Language Model for Continual Learning in Anomaly Detectioncited by 1
  31. 2024How Vision-Language Tasks Benefit from Large Pre-trained Models: A Surveycited by 0
  32. 2025TARS: MinMax Token-Adaptive Preference Strategy for Hallucination Reduction in MLLMscited by 0
  33. 2025KORE: Enhancing Knowledge Injection for Large Multimodal Models via Knowledge-Oriented Controlscited by 0
  34. 2026Continual-NExT: A Unified Comprehension And Generation Continual Learning Frameworkcited by 0
  35. 2026Structural Abstraction as an Inductive Bias for Non-Stationary Language Model Trainingcited by 0
  36. 2026Continual Vision-Language Learning for Remote Sensing: Benchmarking and Analysiscited by 0
  37. 2026Octopus: History-Free Gradient Orthogonalization for Continual Learning in Multimodal Large Language Modelscited by 0
  38. 2026ACE-LoRA: Adaptive Orthogonal Decoupling for Continual Image Editingcited by 0

50 most cited of 60 All 60 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Enhancing Multimodal Continual Instruction Tuning with BranchLoRA
  2. 2024SwitchCIT: Switching for Continual Instruction Tuning
  3. 2026CoRe-MoE: Compact Reusable MoE for Continual Multimodal Instruction Tuning
  4. 2023InstructAlign: High-and-Low Resource Language Alignment via Continual Crosslingual Instruction Tuning
  5. 2025Aligning Instruction Tuning with Pre-training
  6. 2025Towards Alignment-Centric Paradigm: A Survey of Instruction Tuning in Large Language Models
  7. 2023FollowEval: A Multi-Dimensional Benchmark for Assessing the Instruction-Following Capability of Large Language Models
  8. 2024Examining Forgetting in Continual Pre-training of Aligned Large Language Models