# Efficient Prompting for Continual Adaptation to Missing Modalities

arXiv: https://arxiv.org/abs/2503.00528 · HTML: https://mlsanity.com/paper/2503.00528/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 43 references that are in the map.
- First posted on arXiv 1 Mar 2025.

## Builds on (most cited first, 27 of 27)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Learning without Forgetting](https://mlsanity.com/paper/1606.09282/) (2016)
- [iCaRL: Incremental Classifier and Representation Learning](https://mlsanity.com/paper/1611.07725/) (2016)
- [Continual Learning with Deep Generative Replay](https://mlsanity.com/paper/1705.08690/) (2017)
- [Memory Aware Synapses: Learning what (not) to forget](https://mlsanity.com/paper/1711.09601/) (2017)
- [Experience Replay for Continual Learning](https://mlsanity.com/paper/1811.11682/) (2018)
- [Efficient Lifelong Learning with A-GEM](https://mlsanity.com/paper/1812.00420/) (2018)
- [Learning to Prompt for Continual Learning](https://mlsanity.com/paper/2112.08654/) (2021)
- [Dark Experience for General Continual Learning: a Strong, Simple Baseline](https://mlsanity.com/paper/2004.07211/) (2020)
- [PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruning](https://mlsanity.com/paper/1711.05769/) (2017)
- [Overcoming catastrophic forgetting with hard attention to the task](https://mlsanity.com/paper/1801.01423/) (2018)
- [Learn to Grow: A Continual Structure Learning Framework for Overcoming Catastrophic Forgetting](https://mlsanity.com/paper/1904.00310/) (2019)
- [MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2005.03545/) (2020)
- [S-Prompts Learning with Pre-trained Transformers: An Occam's Razor for Domain Incremental Learning](https://mlsanity.com/paper/2207.12819/) (2022)
- [SMIL: Multimodal Learning with Severely Missing Modality](https://mlsanity.com/paper/2103.05677/) (2021)
- [Co2L: Contrastive Continual Learning](https://mlsanity.com/paper/2106.14413/) (2021)
- [Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities](https://mlsanity.com/paper/1812.07809/) (2018)
- [Adversarial Continual Learning](https://mlsanity.com/paper/2003.09553/) (2020)
- [Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysis](https://mlsanity.com/paper/2109.00412/) (2021)
- [Multimodal Prompting with Missing Modalities for Visual Recognition](https://mlsanity.com/paper/2303.03369/) (2023)
- [Classifier-guided Gradient Modulation for Enhanced Multimodal Learning](https://mlsanity.com/paper/2411.01409/) (2024)
- [Semi-supervised Deep Generative Modelling of Incomplete Multi-Modality Emotional Data](https://mlsanity.com/paper/1808.02096/) (2018)
- [Low-rank Prompt Interaction for Continual Vision-Language Retrieval](https://mlsanity.com/paper/2501.14369/) (2025)
- [A Wander Through the Multimodal Landscape: Efficient Transfer Learning via Low-rank Sequence Multimodal Adapter](https://mlsanity.com/paper/2412.08979/) (2024)

## Built on by (most cited first, 8 of 8)

- [Chat-Driven Text Generation and Interaction for Person Retrieval](https://mlsanity.com/paper/2509.12662/) (2025)
- [Reasoning Physical Video Generation with Diffusion Timestep Tokens via Reinforcement Learning](https://mlsanity.com/paper/2504.15932/) (2025)
- [ConceptGuard: Continual Personalized Text-to-Image Generation with Forgetting and Confusion Mitigation](https://mlsanity.com/paper/2503.10358/) (2025)
- [Contrastive Cross-Course Knowledge Tracing via Concept Graph Guided Knowledge Transfer](https://mlsanity.com/paper/2505.13489/) (2025)
- [WorldEdit: Towards Open-World Image Editing with a Knowledge-Informed Benchmark](https://mlsanity.com/paper/2602.07095/) (2026)
- [TAP: Parameter-efficient Task-Aware Prompting for Adverse Weather Removal](https://mlsanity.com/paper/2508.07878/) (2025)
- [DeLo: Dual Decomposed Low-Rank Experts Collaboration for Continual Missing Modality Learning](https://mlsanity.com/paper/2603.01632/) (2026)
- [NeuCME: Toward Dynamic Multimodal Continual Learning via Neural Combinatorics of Multiple Experts](https://mlsanity.com/paper/2609.07009/) (2026)

## Abstract (arXiv)

Missing modality issues are common in real-world applications, arising from factors such as equipment failures and privacy concerns. When fine-tuning pre-trained models on downstream datasets with missing modalities, performance can degrade significantly. Current methods often aggregate various missing cases to train recovery modules or align multimodal features, resulting in suboptimal performance, high computational costs, and the risk of catastrophic forgetting in continual environments where data arrives sequentially. In this paper, we formulate the dynamic missing modality problem as a continual learning task and introduce the continual multimodal missing modality task. To address this challenge efficiently, we introduce three types of prompts: modality-specific, task-aware, and task-specific prompts. These prompts enable the model to learn intra-modality, inter-modality, intra-task, and inter-task features. Furthermore, we propose a contrastive task interaction strategy to explicitly learn prompts correlating different modalities. We conduct extensive experiments on three public datasets, where our method consistently outperforms state-of-the-art approaches.