mlsanityData to 7 Oct 2026

SwitchCIT: Switching for Continual Instruction Tuning

arXiv 2407.11780 ↗First posted 16 Jul 2024cs.CL cs.AIcited by 1

On the map: LLM reasoning → Multimodal reasoning

Abstract (arXiv)

Large language models (LLMs) and multimodal models (MMs) have exhibited impressive capabilities in various domains, particularly in general language understanding and visual reasoning. However, these models, trained on massive data, may not be finely optimized for specific tasks triggered by instructions. Continual instruction tuning is crucial to adapt a large model to evolving tasks and domains, ensuring their effectiveness and relevance across a wide range of applications. In the context of continual instruction tuning, where models are sequentially trained on different tasks, catastrophic forgetting can occur, leading to performance degradation on previously learned tasks. This work addresses the catastrophic forgetting in continual instruction learning through a switching mechanism for routing computations to parameter-efficient tuned models. We demonstrate the effectiveness of our method through experiments on continual instruction tuning of different natural language generation tasks and vision-language tasks. We also showcase the advantages of our proposed method in terms of efficiency, scalability, portability, and privacy preservation.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 36 of its 50 references that are in the map. First posted on arXiv 16 Jul 2024.

Builds on

36 of 50 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023Visual Instruction Tuningcited by 7,792
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  10. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  11. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  12. 2020Learning to summarize from human feedbackcited by 2,562
24 more
  1. 2018Efficient Lifelong Learning with A-GEMcited by 1,044
  2. 2020Dark Experience for General Continual Learning: a Strong, Simple Baselinecited by 895
  3. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  4. 2023A Comprehensive Survey of Continual Learning: Theory, Method and Applicationcited by 860
  5. 2017Lifelong Learning with Dynamically Expandable Networkscited by 689
  6. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  7. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  8. 2019ELI5: Long Form Question Answeringcited by 567
  9. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  10. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  11. 2023LoraHub: Efficient Cross-Task Generalization via Dynamic LoRA Compositioncited by 359
  12. 2021DER: Dynamically Expandable Representation for Class Incremental Learningcited by 355
  13. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 313
  14. 2022DyLoRA: Parameter Efficient Tuning of Pre-trained Models using Dynamic Search-Free Low-Rank Adaptationcited by 229
  15. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 227
  16. 2022Fine-tuned Language Models are Continual Learnerscited by 152
  17. 2020Neural CRF Model for Sentence Alignment in Text Simplificationcited by 106
  18. 2020ASSET: A Dataset for Tuning and Evaluation of Sentence Simplification Models with Multiple Rewriting Transformationscited by 101
  19. 2020Efficient Continual Learning with Modular Networks and Task-Driven Priorscited by 87
  20. 2024Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learningcited by 53
  21. 2021Effective Attention Sheds Light On Interpretabilitycited by 11
  22. 2023A Meta-Learning Perspective on Transformers for Causal Language Modelingcited by 5
  23. 2024Transformer-based Causal Language Models Perform Clusteringcited by 5
  24. 2024Theories of synaptic memory consolidation and intelligent plasticity for continual learningcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Beyond Static Models: An Evolving Framework for Continual Learning in Large Language Models across Training Stagescited by 4

Similar, not cited

Close in wording, no citation either way

  1. 2023Continual Instruction Tuning for Large Multimodal Models
  2. 2025Multimodal Continual Instruction Tuning with Dynamic Gradient Guidance
  3. 2024CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Model
  4. 2024SMoLoRA: Exploring and Defying Dual Catastrophic Forgetting in Continual Visual Instruction Tuning
  5. 2026MAny: Merge Anything for Multimodal Continual Instruction Tuning
  6. 2026CRAM: Centroid-Routing and Adaptive MoE for Multimodal Continual Instruction Tuning
  7. 2025LoRA in LoRA: Towards Parameter-Efficient Architecture Expansion for Continual Visual Instruction Tuning
  8. 2024Continual LLaVA: Continual Instruction Tuning in Large Vision-Language Models