mlsanityData to 7 Oct 2026

ModalPrompt: Towards Efficient Multimodal Continual Instruction Tuning with Dual-Modality Guided Prompt

arXiv 2410.05849 ↗First posted 8 Oct 2024cs.CVcited by 29

On the map: Multimodal LLMs / vision-language models → Prompt learning and adaptation of CLIP

Abstract (arXiv)

Large Multimodal Models (LMMs) exhibit remarkable multi-tasking ability by learning mixed instruction datasets. However, novel tasks would be encountered sequentially in dynamic world, which urges for equipping LMMs with multimodal continual instruction learning (MCIT) ability especially for diverse and challenging generative tasks. Existing MCIT methods do not fully exploit the unique attribute of LMMs and often gain performance at the expense of efficiency. In this paper, we propose a novel prompt learning framework for MCIT to effectively alleviate forgetting of previous knowledge while managing computational complexity with natural image-text supervision. Concretely, we learn prompts for each task and exploit efficient prompt fusion for knowledge transfer and prompt selection for complexity management with dual-modality guidance. Extensive experiments demonstrate that our approach achieves substantial +14.26% performance gain on MCIT benchmarks with remarkable × 1.42 inference speed free from growing computation. Code is available at https://github.com/AuroraZengfh/ModalPrompt.

Key facts. Cited by 29 papers in the map (data through 7 Oct 2026). Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 51 references that are in the map. First posted on arXiv 8 Oct 2024.

Builds on

27 of 51 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Continual Instruction Tuning for Large Multimodal Models27 Nov
  2. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling6 Nov

Other references · 25

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  3. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  4. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  5. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  6. 2016Learning without Forgettingcited by 2,465
  7. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  8. 2019Towards VQA Models That Can Readcited by 1,696
  9. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  10. 2021Learning to Prompt for Continual Learningcited by 938
  11. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 897
  12. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
13 more
  1. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  2. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 635
  3. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  4. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 545
  5. 2022CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learningcited by 446
  6. 2023Orthogonal Subspace Learning for Language Model Continual Learningcited by 296
  7. 2021TransVG: End-to-End Visual Grounding with Transformerscited by 269
  8. 2023Progressive Prompts: Continual Learning for Language Modelscited by 195
  9. 2024Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Modelscited by 54
  10. 2024InsCL: A Data-efficient Continual Learning Paradigm for Fine-tuning Large Language Models with Instructionscited by 54
  11. 2024Multimodal Instruction Tuning with Conditional Mixture of LoRAcited by 36
  12. 2024Rehearsal-Free Modular and Compositional Continual Learning for Language Modelscited by 26
  13. 2023POP: Prompt Of Prompts for Continual Learningcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 24.

29 papers.

Sort:

By area:

  1. 2025Reinforcement Fine-Tuning Naturally Mitigates Forgetting in Continual Post-Trainingcited by 42
  2. 2025Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgettingcited by 16
  3. 2025MCITlib: Multimodal Continual Instruction Tuning Library and Benchmarkcited by 10
  4. 2025When Continue Learning Meets Multimodal Large Language Model: A Surveycited by 8
  5. 2026SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuningcited by 8
  6. 2026FOREVER: Forgetting Curve-Inspired Memory Replay for Language Model Continual Learningcited by 7
  7. 2026Fine-Grained Post-Training Quantization for Large Vision Language Models with Quantization-Aware Integrated Gradientscited by 7
  8. 2025Multimodal Continual Instruction Tuning with Dynamic Gradient Guidancecited by 6
  9. 2026Beyond Static Models: An Evolving Framework for Continual Learning in Large Language Models across Training Stagescited by 4
  10. 2026Prism: A Plug-in Reproducible Infrastructure for Scalable Multimodal Continual Instruction Tuningcited by 3
  11. 2026φ-DPO: Fairness Direct Preference Optimization Approach to Continual Learning in Large Multimodal Modelscited by 2
  12. 2026On Token's Dilemma: Dynamic MoE with Drift-Aware Token Assignment for Continual Learning of Large Vision Language Modelscited by 2
17 more
  1. 2026Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuningcited by 2
  2. 2026DynaTokens: Controlling Token Dynamics for Continual Video-Language Understandingcited by 1
  3. 2026Fine-tuning MLLMs Without Forgetting Is Easier Than You Thinkcited by 1
  4. 2026CL-VISTA: Benchmarking Continual Learning in Video Large Language Modelscited by 1
  5. 2026The Blind Spot of Adaptation: Quantifying and Mitigating Forgetting in Fine-tuned Driving Modelscited by 1
  6. 2026Hidden Forgetting in Continual Multimodal Learning: When Accuracy Survives but Grounding Failscited by 1
  7. 2026AWARe: Mitigating Catastrophic Forgetting via Activation-Weighted Adaptive REtentioncited by 1
  8. 2026CRAM: Centroid-Routing and Adaptive MoE for Multimodal Continual Instruction Tuningcited by 0
  9. 2026ProtoAda: Prototype-Guided Adaptive Adapter Expansion and Geometric Consolidation for Multimodal Continual Instruction Tuningcited by 0
  10. 2026ECA: Efficient Continual Alignment for Open-Ended Image-to-Text Generationcited by 0
  11. 2026Curvature-Guided Mixing for MLLM Adaptationcited by 0
  12. 2026SiGMA: Sign-Guided Merging and Adaptation for Multimodal Continual Instruction Tuningcited by 0
  13. 2026In-Context Collapse in Vision-Language Models and How to Mitigate it?cited by 0
  14. 2026CoRe-MoE: Compact Reusable MoE for Continual Multimodal Instruction Tuningcited by 0
  15. 2026Hyper-LLaVA: Hyperbolic Uncertainty-aware Modality-Balanced Routing for Multimodal Continual Instruction Tuningcited by 0
  16. 2026UniAR: A Unified Framework for Autism Recognition Enhanced by Multi-View Prompt Learningcited by 0
  17. 2026ReCAP: Retrieval-Guided Capability Reuse for Multimodal Continual Instruction Tuningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026MAny: Merge Anything for Multimodal Continual Instruction Tuning
  2. 2024Beyond Anti-Forgetting: Multimodal Continual Instruction Tuning with Positive Forward Transfer
  3. 2024Modality-Inconsistent Continual Learning of Multimodal Large Language Models
  4. 2025Multi-Modal Continual Learning via Cross-Modality Adapters and Representation Alignment with Knowledge Preservation
  5. 2026Progressive Multimodal Alignment for Continual Instruction Tuning
  6. 2025Merge then Realign: Simple and Effective Modality-Incremental Continual Learning for Multimodal LLMs
  7. 2025Enhancing Multimodal Continual Instruction Tuning with BranchLoRA
  8. 2024SwitchCIT: Switching for Continual Instruction Tuning