mlsanityData to 14 Oct 2026

Federated Continual Instruction Tuning

arXiv 2503.12897 ↗First posted 17 Mar 2025cs.LG cs.AIcited by 12

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

A vast amount of instruction tuning data is crucial for the impressive performance of Large Multimodal Models (LMMs), but the associated computational costs and data collection demands during supervised fine-tuning make it impractical for most researchers. Federated learning (FL) has the potential to leverage all distributed data and training resources to reduce the overhead of joint training. However, most existing methods assume a fixed number of tasks, while in real-world scenarios, clients continuously encounter new knowledge and often struggle to retain old tasks due to memory constraints. In this work, we introduce the Federated Continual Instruction Tuning (FCIT) benchmark to model this real-world challenge. Our benchmark includes two realistic scenarios, encompassing four different settings and twelve carefully curated instruction tuning datasets. To address the challenges posed by FCIT, we propose dynamic knowledge organization to effectively integrate updates from different tasks during training and subspace selective activation to allocate task-specific output during inference. Extensive experimental results demonstrate that our proposed method significantly enhances model performance across varying levels of data heterogeneity and catastrophic forgetting. Code and dataset are released at https://github.com/Ghy0501/FCIT.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 53 of its 69 references that are in the map. First posted on arXiv 17 Mar 2025.

Builds on

53 of 69 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Continual LLaVA: Continual Instruction Tuning in Large Vision-Language Models4 Nov
  2. 2024CoIN: A Benchmark of Continual Instruction tuNing for Multimodel Large Language Model13 Mar

Other references · 51

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  7. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  8. 2023Mistral 7Bcited by 3,164
  9. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,876
  10. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  11. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,711
  12. 2016Learning without Forgettingcited by 2,496
38 more
  1. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  2. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,538
  3. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  4. 2020Adaptive Federated Optimizationcited by 1,046
  5. 2021Learning to Prompt for Continual Learningcited by 944
  6. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  7. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 851
  8. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 733
  9. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  10. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 568
  11. 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 352
  12. 2022Dynamic Prompt Learning via Policy Gradient for Semi-structured Mathematical Reasoningcited by 350
  13. 2018DVQA: Understanding Data Visualizations via Question Answeringcited by 340
  14. 2017FigureQA: An Annotated Figure Dataset for Visual Reasoningcited by 319
  15. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 312
  16. 2021IconQA: A New Benchmark for Abstract Diagram Understanding and Visual Language Reasoningcited by 267
  17. 2014Towards Open World Recognitioncited by 226
  18. 2023Towards Building the Federated GPT: Federated Instruction Tuningcited by 185
  19. 2020Federated Continual Learning with Weighted Inter-client Transfercited by 161
  20. 2024FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptationscited by 146
  21. 2022Federated Learning from Pre-Trained Models: A Contrastive Learning Approachcited by 142
  22. 2024Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Modelscited by 139
  23. 2024OpenFedLLM: Training Large Language Models on Decentralized Private Data via Federated Learningcited by 128
  24. 2022Federated Class-Incremental Learningcited by 127
  25. 2022CLEVR-Math: A Dataset for Compositional Language, Visual and Mathematical Reasoningcited by 120
  26. 2022Super-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoningcited by 92
  27. 2023TARGET: Federated Class-Continual Learning via Exemplar-Free Distillationcited by 65
  28. 2023FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learningcited by 60
  29. 2023Federated Continual Learning via Knowledge Fusion: A Surveycited by 53
  30. 2024Mixture-of-Subspaces in Low-Rank Adaptationcited by 47
  31. 2023A Data-Free Approach to Mitigate Catastrophic Forgetting in Federated Class Incremental Learning for Vision Taskscited by 41
  32. 2020Towards Robust Pattern Recognition: A Reviewcited by 31
  33. 2024The Future of Large Language Model Pre-training is Federatedcited by 30
  34. 2023Asynchronous Federated Continual Learningcited by 29
  35. 2023Learnability and Algorithm for Continual Learningcited by 28
  36. 2024Open-world machine learning: A review and new outlookscited by 27
  37. 2024PILoRA: Prototype Guided Incremental LoRA for Federated Class-Incremental Learningcited by 19
  38. 2024Local-Prompt: Extensible Local Prompts for Few-Shot Out-of-Distribution Detectioncited by 15

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 6.

12 papers.

  1. 2025MLLM-CL: Continual Learning for Multimodal Large Language Modelscited by 23
  2. 2025Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgettingcited by 18
  3. 2025MCITlib: Multimodal Continual Instruction Tuning Library and Benchmarkcited by 13
  4. 2025Continual Learning for Generative AI: From LLMs to MLLMs and Beyondcited by 12
  5. 2025Multimodal Continual Instruction Tuning with Dynamic Gradient Guidancecited by 7
  6. 2026Model-Dowser: Data-Free Importance Probing to Mitigate Catastrophic Forgetting in Multimodal Large Language Modelscited by 2
  7. 2026φ-DPO: Fairness Direct Preference Optimization Approach to Continual Learning in Large Multimodal Modelscited by 2
  8. 2026CL-VISTA: Benchmarking Continual Learning in Video Large Language Modelscited by 2
  9. 2025Towards Efficient and General-Purpose Few-Shot Misclassification Detection for Vision-Language Modelscited by 1
  10. 2026SiGMA: Sign-Guided Merging and Adaptation for Multimodal Continual Instruction Tuningcited by 0
  11. 2026PaLoRA: Paced Low-Rank Adaptation for Continual Learningcited by 0
  12. 2026Monarch-MoE: Structured Experts for Multimodal Continual Instruction Tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Pilot: Building the Federated Multimodal Instruction Tuning Framework
  2. 2023Continual Instruction Tuning for Large Multimodal Models
  3. 2025Enhancing Multimodal Continual Instruction Tuning with BranchLoRA
  4. 2024SwitchCIT: Switching for Continual Instruction Tuning
  5. 2024Adapt-∞: Scalable Continual Multimodal Instruction Tuning via Dynamic Data Selection
  6. 2026Continual Learning with Elastic Regularization and Synthetic Replay for Federated MLLM Fine-Tuning
  7. 2026UniFed-VLM: Federated Instruction Tuning for Vision-Language Models with Multiple Heterogeneity
  8. 2026ReCAP: Retrieval-Guided Capability Reuse for Multimodal Continual Instruction Tuning