# ModalPrompt: Towards Efficient Multimodal Continual Instruction Tuning with Dual-Modality Guided Prompt

arXiv: https://arxiv.org/abs/2410.05849 · HTML: https://mlsanity.com/paper/2410.05849/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Prompt learning and adaptation of CLIP, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 27 of its 51 references that are in the map.
- First posted on arXiv 8 Oct 2024.

## Placed in

- Multimodal LLMs / vision-language models / [Prompt learning and adaptation of CLIP](https://mlsanity.com/mllm/prompt-learning-and-adaptation-of-clip/)

## Builds on (most cited first, 27 of 27)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuning](https://mlsanity.com/paper/2305.06500/) (2023)
- [Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answering](https://mlsanity.com/paper/1612.00837/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [Learning to Prompt for Continual Learning](https://mlsanity.com/paper/2112.08654/) (2021)
- [Generation and Comprehension of Unambiguous Object Descriptions](https://mlsanity.com/paper/1511.02283/) (2015)
- [VizWiz Grand Challenge: Answering Visual Questions from Blind People](https://mlsanity.com/paper/1802.08218/) (2018)
- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://mlsanity.com/paper/2403.14608/) (2024)
- [DualPrompt: Complementary Prompting for Rehearsal-free Continual Learning](https://mlsanity.com/paper/2204.04799/) (2022)
- [FILIP: Fine-grained Interactive Language-Image Pre-Training](https://mlsanity.com/paper/2111.07783/) (2021)
- [mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaboration](https://mlsanity.com/paper/2311.04257/) (2023)
- [CODA-Prompt: COntinual Decomposed Attention-based Prompting for Rehearsal-Free Continual Learning](https://mlsanity.com/paper/2211.13218/) (2022)
- [Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modeling](https://mlsanity.com/paper/2111.03930/) (2021)
- [Orthogonal Subspace Learning for Language Model Continual Learning](https://mlsanity.com/paper/2310.14152/) (2023)
- [TransVG: End-to-End Visual Grounding with Transformers](https://mlsanity.com/paper/2104.08541/) (2021)
- [Progressive Prompts: Continual Learning for Language Models](https://mlsanity.com/paper/2301.12314/) (2023)
- [InsCL: A Data-efficient Continual Learning Paradigm for Fine-tuning Large Language Models with Instructions](https://mlsanity.com/paper/2403.11435/) (2024)
- [Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Models](https://mlsanity.com/paper/2402.12048/) (2024)
- [Continual Instruction Tuning for Large Multimodal Models](https://mlsanity.com/paper/2311.16206/) (2023)
- [Multimodal Instruction Tuning with Conditional Mixture of LoRA](https://mlsanity.com/paper/2402.15896/) (2024)
- [Rehearsal-Free Modular and Compositional Continual Learning for Language Models](https://mlsanity.com/paper/2404.00790/) (2024)
- [POP: Prompt Of Prompts for Continual Learning](https://mlsanity.com/paper/2306.08200/) (2023)

## Built on by (most cited first, 30 of 30)

- [Reinforcement Fine-Tuning Naturally Mitigates Forgetting in Continual Post-Training](https://mlsanity.com/paper/2507.05386/) (2025)
- [Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting](https://mlsanity.com/paper/2508.04227/) (2025)
- [MCITlib: Multimodal Continual Instruction Tuning Library and Benchmark](https://mlsanity.com/paper/2508.07307/) (2025)
- [When Continue Learning Meets Multimodal Large Language Model: A Survey](https://mlsanity.com/paper/2503.01887/) (2025)
- [FOREVER: Forgetting Curve-Inspired Memory Replay for Language Model Continual Learning](https://mlsanity.com/paper/2601.03938/) (2026)
- [SAME: Stabilized Mixture-of-Experts for Multimodal Continual Instruction Tuning](https://mlsanity.com/paper/2602.01990/) (2026)
- [Fine-Grained Post-Training Quantization for Large Vision Language Models with Quantization-Aware Integrated Gradients](https://mlsanity.com/paper/2603.17809/) (2026)
- [Multimodal Continual Instruction Tuning with Dynamic Gradient Guidance](https://mlsanity.com/paper/2511.15164/) (2025)
- [Beyond Static Models: An Evolving Framework for Continual Learning in Large Language Models across Training Stages](https://mlsanity.com/paper/2603.12658/) (2026)
- [Prism: A Plug-in Reproducible Infrastructure for Scalable Multimodal Continual Instruction Tuning](https://mlsanity.com/paper/2605.26110/) (2026)
- [φ-DPO: Fairness Direct Preference Optimization Approach to Continual Learning in Large Multimodal Models](https://mlsanity.com/paper/2602.22601/) (2026)
- [Fine-tuning MLLMs Without Forgetting Is Easier Than You Think](https://mlsanity.com/paper/2603.14493/) (2026)
- [On Token's Dilemma: Dynamic MoE with Drift-Aware Token Assignment for Continual Learning of Large Vision Language Models](https://mlsanity.com/paper/2603.27481/) (2026)
- [CL-VISTA: Benchmarking Continual Learning in Video Large Language Models](https://mlsanity.com/paper/2604.00677/) (2026)
- [Dynamic Cross-Modal Prompt Generation for Multimodal Continual Instruction Tuning](https://mlsanity.com/paper/2605.10765/) (2026)
- [DynaTokens: Controlling Token Dynamics for Continual Video-Language Understanding](https://mlsanity.com/paper/2603.06662/) (2026)
- [The Blind Spot of Adaptation: Quantifying and Mitigating Forgetting in Fine-tuned Driving Models](https://mlsanity.com/paper/2604.04857/) (2026)
- [Hidden Forgetting in Continual Multimodal Learning: When Accuracy Survives but Grounding Fails](https://mlsanity.com/paper/2607.02020/) (2026)
- [AWARe: Mitigating Catastrophic Forgetting via Activation-Weighted Adaptive REtention](https://mlsanity.com/paper/2608.11758/) (2026)
- [CRAM: Centroid-Routing and Adaptive MoE for Multimodal Continual Instruction Tuning](https://mlsanity.com/paper/2606.02502/) (2026)
- [ProtoAda: Prototype-Guided Adaptive Adapter Expansion and Geometric Consolidation for Multimodal Continual Instruction Tuning](https://mlsanity.com/paper/2606.02576/) (2026)
- [ECA: Efficient Continual Alignment for Open-Ended Image-to-Text Generation](https://mlsanity.com/paper/2606.12633/) (2026)
- [Curvature-Guided Mixing for MLLM Adaptation](https://mlsanity.com/paper/2606.24963/) (2026)
- [SiGMA: Sign-Guided Merging and Adaptation for Multimodal Continual Instruction Tuning](https://mlsanity.com/paper/2607.20511/) (2026)
- [In-Context Collapse in Vision-Language Models and How to Mitigate it?](https://mlsanity.com/paper/2608.02830/) (2026)
- [CoRe-MoE: Compact Reusable MoE for Continual Multimodal Instruction Tuning](https://mlsanity.com/paper/2608.27867/) (2026)
- [Hyper-LLaVA: Hyperbolic Uncertainty-aware Modality-Balanced Routing for Multimodal Continual Instruction Tuning](https://mlsanity.com/paper/2609.13742/) (2026)
- [UniAR: A Unified Framework for Autism Recognition Enhanced by Multi-View Prompt Learning](https://mlsanity.com/paper/2609.31298/) (2026)
- [ReCAP: Retrieval-Guided Capability Reuse for Multimodal Continual Instruction Tuning](https://mlsanity.com/paper/2609.37889/) (2026)
- [PaLoRA: Paced Low-Rank Adaptation for Continual Learning](https://mlsanity.com/paper/2610.04226/) (2026)

## Abstract (arXiv)

Large Multimodal Models (LMMs) exhibit remarkable multi-tasking ability by learning mixed instruction datasets. However, novel tasks would be encountered sequentially in dynamic world, which urges for equipping LMMs with multimodal continual instruction learning (MCIT) ability especially for diverse and challenging generative tasks. Existing MCIT methods do not fully exploit the unique attribute of LMMs and often gain performance at the expense of efficiency. In this paper, we propose a novel prompt learning framework for MCIT to effectively alleviate forgetting of previous knowledge while managing computational complexity with natural image-text supervision. Concretely, we learn prompts for each task and exploit efficient prompt fusion for knowledge transfer and prompt selection for complexity management with dual-modality guidance. Extensive experiments demonstrate that our approach achieves substantial +14.26% performance gain on MCIT benchmarks with remarkable × 1.42 inference speed free from growing computation. Code is available at https://github.com/AuroraZengfh/ModalPrompt.