# Towards Robust Multimodal Prompting With Missing Modalities

arXiv: https://arxiv.org/abs/2312.15890 · HTML: https://mlsanity.com/paper/2312.15890/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 21 references that are in the map.
- First posted on arXiv 26 Dec 2023.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [Learning to Prompt for Continual Learning](https://mlsanity.com/paper/2112.08654/) (2021)
- [Multimodal Few-Shot Learning with Frozen Language Models](https://mlsanity.com/paper/2106.13884/) (2021)
- [Gated Multimodal Units for Information Fusion](https://mlsanity.com/paper/1702.01992/) (2017)
- [Are Multimodal Transformers Robust to Missing Modality?](https://mlsanity.com/paper/2204.05454/) (2022)
- [Multimodal Prompting with Missing Modalities for Visual Recognition](https://mlsanity.com/paper/2303.03369/) (2023)
- [Towards Good Practices for Missing Modality Robust Action Recognition](https://mlsanity.com/paper/2211.13916/) (2022)
- [MM-ViT: Multi-Modal Video Transformer for Compressed Video Action Recognition](https://mlsanity.com/paper/2108.09322/) (2021)
- [Deep Multimodal Feature Analysis for Action Recognition in RGB+D Videos](https://mlsanity.com/paper/1603.07120/) (2016)

## Built on by (most cited first, 16 of 16)

- [Deep Multimodal Learning with Missing Modality: A Survey](https://mlsanity.com/paper/2409.07825/) (2024)
- [Retrieval-Augmented Dynamic Prompt Tuning for Incomplete Multimodal Learning](https://mlsanity.com/paper/2501.01120/) (2025)
- [MMP: Towards Robust Multi-Modal Learning with Masked Modality Projection](https://mlsanity.com/paper/2410.03010/) (2024)
- [Towards Unified Vision-Language Models with Incomplete Multi-Modal Inputs](https://mlsanity.com/paper/2605.27894/) (2026)
- [Reconstruct before Query: Continual Missing Modality Learning with Decomposed Prompt Collaboration](https://mlsanity.com/paper/2403.11373/) (2024)
- [MoRA: LoRA Guided Multi-Modal Disease Diagnosis with Missing Modality](https://mlsanity.com/paper/2408.09064/) (2024)
- [Enhancing Multimodal Sentiment Analysis for Missing Modality through Self-Distillation and Unified Modality Cross-Attention](https://mlsanity.com/paper/2410.15029/) (2024)
- [EPE-P: Evidence-based Parameter-efficient Prompting for Multimodal Learning with Missing Modalities](https://mlsanity.com/paper/2412.17677/) (2024)
- [Robust Multimodal Learning via Cross-Modal Proxy Tokens](https://mlsanity.com/paper/2501.17823/) (2025)
- [COMPASS: Fusion-Matched Supervision for Missing-Modality Human Sensing](https://mlsanity.com/paper/2604.02056/) (2026)
- [MDA: An Interpretable and Scalable Multi-Modal Fusion under Missing Modalities and Intrinsic Noise Conditions](https://mlsanity.com/paper/2406.10569/) (2024)
- [Are you SURE? Enhancing Multimodal Pretraining with Missing Modalities through Uncertainty Estimation](https://mlsanity.com/paper/2504.13465/) (2025)
- [DPL: Decoupled Prototype Learning for Enhancing Robustness of Vision-Language Transformers to Missing Modalities](https://mlsanity.com/paper/2505.08283/) (2025)
- [Buffer replay enhances the robustness of multimodal learning under missing-modality](https://mlsanity.com/paper/2511.23070/) (2025)
- [Enhancing Foundation VLM Robustness to Missing Modality: Scalable Diffusion for Bi-directional Feature Restoration](https://mlsanity.com/paper/2602.03151/) (2026)
- [AOEPT: Breaking the Implicit Modality-Reduction Bottleneck in Modality-Missing Prompt Tuning](https://mlsanity.com/paper/2605.24816/) (2026)

## Abstract (arXiv)

Recently, multimodal prompting, which introduces learnable missing-aware prompts for all missing modality cases, has exhibited impressive performance. However, it encounters two critical issues: 1) The number of prompts grows exponentially as the number of modalities increases; and 2) It lacks robustness in scenarios with different missing modality settings between training and inference. In this paper, we propose a simple yet effective prompt design to address these challenges. Instead of using missing-aware prompts, we utilize prompts as modality-specific tokens, enabling them to capture the unique characteristics of each modality. Furthermore, our prompt design leverages orthogonality between prompts as a key element to learn distinct information across different modalities and promote diversity in the learned representations. Extensive experiments demonstrate that our prompt design enhances both performance and robustness while reducing the number of prompts.