mlsanityData to 7 Oct 2026

Multimodal Prompting with Missing Modalities for Visual Recognition

arXiv 2303.03369 ↗First posted 6 Mar 2023cs.CVcited by 121

Not on the map. Why?

Abstract (arXiv)

In this paper, we tackle two challenges in multimodal learning for visual recognition: 1) when missing-modality occurs either during training or testing in real-world situations; and 2) when the computation resources are not available to finetune on heavy transformer models. To this end, we propose to utilize prompt learning and mitigate the above two challenges together. Specifically, our modality-missing-aware prompts can be plugged into multimodal transformers to handle general missing-modality cases, while only requiring less than 1% learnable parameters compared to training the entire model. We further explore the effect of different prompt configurations and analyze the robustness to missing modality. Extensive experiments are conducted to show the effectiveness of our prompt learning framework that improves the performance under various missing-modality cases, while alleviating the requirement of heavy model re-training. Code is available.

Key facts. Cited by 121 papers in the map (data through 7 Oct 2026). Builds on 39 of its 48 references that are in the map. First posted on arXiv 6 Mar 2023.

Builds on

39 of 48 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  9. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  10. 2022Visual Prompt Tuningcited by 1,674
  11. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  12. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
27 more
  1. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  2. 2021Learning to Prompt for Continual Learningcited by 938
  3. 2022MaPLe: Multi-modal Prompt Learningcited by 685
  4. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
  5. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  6. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 636
  7. 2022DualPrompt: Complementary Prompting for Rehearsal-free Continual Learningcited by 635
  8. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  9. 2020Multi-modal Transformer for Video Retrievalcited by 395
  10. 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 254
  11. 2017Gated Multimodal Units for Information Fusioncited by 244
  12. 2021SMIL: Multimodal Learning with Severely Missing Modalitycited by 243
  13. 2022Are Multimodal Transformers Robust to Missing Modality?cited by 183
  14. 2023Pic2Word: Mapping Pictures to Words for Zero-shot Composed Image Retrievalcited by 155
  15. 2018Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalitiescited by 149
  16. 2021End-to-End Referring Video Object Segmentation with Multimodal Transformerscited by 135
  17. 2022What Language Model to Train if You Have One Million GPU Hours?cited by 92
  18. 2022FormNet: Structural Encoding beyond Sequential Modeling in Form Document Information Extractioncited by 64
  19. 2022Prompting for Multi-Modal Trackingcited by 58
  20. 2020A Transformer-based joint-encoding for Emotion Recognition and Sentiment Analysiscited by 46
  21. 2020TransModality: An End2End Fusion Method with Transformer for Multimodal Sentiment Analysiscited by 40
  22. 2022Modular and Parameter-Efficient Multimodal Fusion with Promptingcited by 35
  23. 2022Robust Fine-Tuning of Deep Neural Networks with Hessian-based Generalization Guaranteescited by 34
  24. 2022Tag-assisted Multimodal Sentiment Analysis under Uncertain Missing Modalitiescited by 28
  25. 2022Prefix Conditioning Unifies Language and Label Supervisioncited by 13
  26. 2020Do Fine-tuned Commonsense Language Models Really Generalize?cited by 8
  27. 2021MEmoBERT: Pre-training Model with Prompt-based Learning for Multimodal Emotion Recognitioncited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 12; 2024: 42; 2025: 30; 2026: 37.

121 papers.

Sort:

By area:

  1. 2024Deep Multimodal Learning with Missing Modality: A Surveycited by 89
  2. 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directionscited by 77
  3. 2023Single-Model and Any-Modality for Video Object Trackingcited by 64
  4. 2023Robust Multimodal Learning with Missing Modalities via Parameter-Efficient Adaptationcited by 27
  5. 2024Multimodal Prompt Learning with Missing Modalities for Sentiment Analysis and Emotion Recognitioncited by 26
  6. 2025SimMLM: A Simple Framework for Multi-modal Learning with Missing Modalitycited by 22
  7. 2025Prompt-based Adaptation in Large-scale Vision Models: A Surveycited by 21
  8. 2024Missing Modality Prediction for Unpaired Multimodal Learning via Joint Embedding of Unimodal Modelscited by 20
  9. 2024All in One Framework for Multimodal Re-identification in the Wildcited by 18
  10. 2023Modality-missing RGBT Tracking: Invertible Prompt Learning and High-quality Benchmarkscited by 17
  11. 2024Deep Correlated Prompting for Visual Recognition with Missing Modalitiescited by 17
  12. 2023Towards Robust Multimodal Prompting With Missing Modalitiescited by 16
38 more
  1. 2024Quantifying and Enhancing Multi-modal Robustness with Modality Preferencecited by 16
  2. 2025Retrieval-Augmented Dynamic Prompt Tuning for Incomplete Multimodal Learningcited by 15
  3. 2023Delving into Multimodal Prompting for Fine-grained Visual Classificationcited by 14
  4. 2025Robust Multimodal Survival Prediction with the Latent Differentiation Conditional Variational AutoEncodercited by 13
  5. 2024A Multimodal Single-Branch Embedding Network for Recommendation in Cold-Start and Missing Modality Scenarioscited by 12
  6. 2024Fourier Prompt Tuning for Modality-Incomplete Scene Segmentationcited by 11
  7. 2024MuAP: Multi-step Adaptive Prompt Learning for Vision-Language Model with Missing Modalitycited by 10
  8. 2024MMP: Towards Robust Multi-Modal Learning with Masked Modality Projectioncited by 10
  9. 2025Distilled Prompt Learning for Incomplete Multimodal Survival Predictioncited by 10
  10. 2023Visual Prompt Flexible-Modal Face Anti-Spoofingcited by 9
  11. 2023Unified-modal Salient Object Detection via Adaptive Prompt Learningcited by 9
  12. 2025Knowledge Bridger: Towards Training-free Missing Modality Completioncited by 9
  13. 2025See-Saw Modality Balance: See Gradient, and Sew Impaired Vision-Language Balance to Mitigate Dominant Modality Biascited by 9
  14. 2025Reducing Unimodal Bias in Multi-Modal Semantic Segmentation with Multi-Scale Functional Entropy Regularizationcited by 9
  15. 2023ChatGPT-Like Large-Scale Foundation Models for Prognostics and Health Management: A Survey and Roadmapscited by 8
  16. 2024Recurrent Aligned Network for Generalized Pedestrian Trajectory Predictioncited by 8
  17. 2025Efficient Prompting for Continual Adaptation to Missing Modalitiescited by 8
  18. 2025Synergistic Prompting for Robust Visual Recognition with Missing Modalitiescited by 8
  19. 2024Exploring Missing Modality in Multimodal Egocentric Datasetscited by 7
  20. 2024An Empirical Categorization of Prompting Techniques for Large Language Models: A Practitioner's Guidecited by 7
  21. 2024GTP-4o: Modality-prompted Heterogeneous Graph Learning for Omni-modal Biomedical Representationcited by 7
  22. 2024MMBind: Unleashing the Potential of Distributed and Heterogeneous Data for Multimodal Learning in IoTcited by 6
  23. 2024Bag of Tricks for Multimodal AutoML with Image, Text, and Tabular Datacited by 6
  24. 2025ReID5o: Achieving Omni Multi-modal Person Re-identification in a Single Modelcited by 6
  25. 2023RGBT Tracking via Progressive Fusion Transformer with Dynamically Guided Learningcited by 5
  26. 2023Zero-Shot and Few-Shot Video Question Answering with Multi-Modal Promptscited by 5
  27. 2024Multimodal Informative ViT: Information Aggregation and Distribution for Hyperspectral and LiDAR Classificationcited by 5
  28. 2024Borrowing Treasures from Neighbors: In-Context Learning for Multimodal Learning with Missing Modalities and Data Scarcitycited by 5
  29. 2024Meta-Learned Modality-Weighted Knowledge Distillation for Robust Multi-Modal Learning with Missing Datacited by 5
  30. 2024CLIP meets DINO for Tuning Zero-Shot Classifier using Unlabeled Image Collectionscited by 5
  31. 2025Foundation Models for CPS-IoT: Opportunities and Challengescited by 5
  32. 2025A Similarity Paradigm Through Textual Regularization Without Forgettingcited by 5
  33. 2025UniMoCo: Unified Modality Completion for Robust Multi-Modal Embeddingscited by 5
  34. 2025Learning Contrastive Multimodal Fusion with Improved Modality Dropout for Disease Detection and Predictioncited by 5
  35. 2026MMLoP: Multi-Modal Low-Rank Prompting for Efficient Vision-Language Adaptationcited by 5
  36. 2023Towards Unified AI Drug Discovery with Multiple Knowledge Modalitiescited by 4
  37. 2023A Survey on Image-text Multimodal Modelscited by 4
  38. 2024Toward Robust Multimodal Learning using Multimodal Foundational Modelscited by 4

50 most cited of 121 All 121 (JSON)