# Retrieval-Augmented Dynamic Prompt Tuning for Incomplete Multimodal Learning

arXiv: https://arxiv.org/abs/2501.01120 · HTML: https://mlsanity.com/paper/2501.01120/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 14 of its 30 references that are in the map.
- First posted on arXiv 2 Jan 2025.

## Placed in

- Retrieval-augmented generation / [Multimodal RAG](https://mlsanity.com/rag/multimodal-rag/)

## Builds on (most cited first, 14 of 14)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [ViLT: Vision-and-Language Transformer Without Convolution or Region Supervision](https://mlsanity.com/paper/2102.03334/) (2021)
- [The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes](https://mlsanity.com/paper/2005.04790/) (2020)
- [SMIL: Multimodal Learning with Severely Missing Modality](https://mlsanity.com/paper/2103.05677/) (2021)
- [Are Multimodal Transformers Robust to Missing Modality?](https://mlsanity.com/paper/2204.05454/) (2022)
- [Multi-modal Learning with Missing Modality via Shared-Specific Feature Modelling](https://mlsanity.com/paper/2307.14126/) (2023)
- [Multimodal Prompting with Missing Modalities for Visual Recognition](https://mlsanity.com/paper/2303.03369/) (2023)
- [Filter-enhanced MLP is All You Need for Sequential Recommendation](https://mlsanity.com/paper/2202.13556/) (2022)
- [Towards Good Practices for Missing Modality Robust Action Recognition](https://mlsanity.com/paper/2211.13916/) (2022)
- [CLIPSyntel: CLIP and LLM Synergy for Multimodal Question Summarization in Healthcare](https://mlsanity.com/paper/2312.11541/) (2023)
- [Correlation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalities](https://mlsanity.com/paper/2404.16456/) (2024)
- [A Survey of Information Cascade Analysis: Models, Predictions, and Recent Advances](https://mlsanity.com/paper/2005.11041/) (2020)
- [Exploiting modality-invariant feature for robust multimodal emotion recognition with missing modalities](https://mlsanity.com/paper/2210.15359/) (2022)
- [Towards Robust Multimodal Prompting With Missing Modalities](https://mlsanity.com/paper/2312.15890/) (2023)

## Built on by (most cited first, 15 of 15)

- [Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation](https://mlsanity.com/paper/2502.08826/) (2025)
- [Synergistic Prompting for Robust Visual Recognition with Missing Modalities](https://mlsanity.com/paper/2507.07802/) (2025)
- [TAMEing Long Contexts in Personalization: Towards Training-Free and State-Aware MLLM Personalized Assistant](https://mlsanity.com/paper/2512.21616/) (2025)
- [WISER: Wider Search, Deeper Thinking, and Adaptive Fusion for Training-Free Zero-Shot Composed Image Retrieval](https://mlsanity.com/paper/2602.23029/) (2026)
- [MM-StanceDet: Retrieval-Augmented Multi-modal Multi-agent Stance Detection](https://mlsanity.com/paper/2604.27934/) (2026)
- [Empowering Multimodal LLMs with External Tools: A Comprehensive Survey](https://mlsanity.com/paper/2508.10955/) (2025)
- [RAID: Retrieval-Augmented Anomaly Detection](https://mlsanity.com/paper/2602.19611/) (2026)
- [OC-Distill: Ontology-aware Contrastive Learning with Cross-Modal Distillation for ICU Risk Prediction](https://mlsanity.com/paper/2604.16878/) (2026)
- [Re-coding for Uncertainties: Edge-awareness Semantic Concordance for Resilient Event-RGB Segmentation](https://mlsanity.com/paper/2511.08269/) (2025)
- [E2PL: Effective and Efficient Prompt Learning for Incomplete Multi-view Multi-Label Class Incremental Learning](https://mlsanity.com/paper/2601.17076/) (2026)
- [Enhancing Foundation VLM Robustness to Missing Modality: Scalable Diffusion for Bi-directional Feature Restoration](https://mlsanity.com/paper/2602.03151/) (2026)
- [AOEPT: Breaking the Implicit Modality-Reduction Bottleneck in Modality-Missing Prompt Tuning](https://mlsanity.com/paper/2605.24816/) (2026)
- [C2MOE: Consistency and Complementarity-guided Mixture of Experts for Incomplete Multimodal Emotion Learning](https://mlsanity.com/paper/2608.04013/) (2026)
- [I Don't Miss You, but I Do: Self-Explanation Faithfulness of Modality Missingness in Vision-Language Models](https://mlsanity.com/paper/2609.07596/) (2026)
- [PACER: Progressive Availability-Conditioned Evidence Routing for Radiology Report Generation under Incomplete Clinical Context](https://mlsanity.com/paper/2609.34487/) (2026)

## Abstract (arXiv)

Multimodal learning with incomplete modality is practical and challenging. Recently, researchers have focused on enhancing the robustness of pre-trained MultiModal Transformers (MMTs) under missing modality conditions by applying learnable prompts. However, these prompt-based methods face several limitations: (1) incomplete modalities provide restricted modal cues for task-specific inference, (2) dummy imputation for missing content causes information loss and introduces noise, and (3) static prompts are instance-agnostic, offering limited knowledge for instances with various missing conditions. To address these issues, we propose RAGPT, a novel Retrieval-AuGmented dynamic Prompt Tuning framework. RAGPT comprises three modules: (I) the multi-channel retriever, which identifies similar instances through a within-modality retrieval strategy, (II) the missing modality generator, which recovers missing information using retrieved contexts, and (III) the context-aware prompter, which captures contextual knowledge from relevant instances and generates dynamic prompts to largely enhance the MMT's robustness. Extensive experiments conducted on three real-world datasets show that RAGPT consistently outperforms all competitive baselines in handling incomplete modality problems. The code of our work and prompt-based baselines is available at https://github.com/Jian-Lang/RAGPT.