mlsanityData to 7 Oct 2026

Retrieval-Augmented Dynamic Prompt Tuning for Incomplete Multimodal Learning

arXiv 2501.01120 ↗First posted 2 Jan 2025cs.CV cs.AIcited by 15

On the map: Retrieval-augmented generation → Multimodal RAG

Abstract (arXiv)

Multimodal learning with incomplete modality is practical and challenging. Recently, researchers have focused on enhancing the robustness of pre-trained MultiModal Transformers (MMTs) under missing modality conditions by applying learnable prompts. However, these prompt-based methods face several limitations: (1) incomplete modalities provide restricted modal cues for task-specific inference, (2) dummy imputation for missing content causes information loss and introduces noise, and (3) static prompts are instance-agnostic, offering limited knowledge for instances with various missing conditions. To address these issues, we propose RAGPT, a novel Retrieval-AuGmented dynamic Prompt Tuning framework. RAGPT comprises three modules: (I) the multi-channel retriever, which identifies similar instances through a within-modality retrieval strategy, (II) the missing modality generator, which recovers missing information using retrieved contexts, and (III) the context-aware prompter, which captures contextual knowledge from relevant instances and generates dynamic prompts to largely enhance the MMT's robustness. Extensive experiments conducted on three real-world datasets show that RAGPT consistently outperforms all competitive baselines in handling incomplete modality problems. The code of our work and prompt-based baselines is available at https://github.com/Jian-Lang/RAGPT.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Multimodal RAG, one of 15 directions in Retrieval-augmented generation. Builds on 14 of its 30 references that are in the map. First posted on arXiv 2 Jan 2025.

Builds on

14 of 30 references.

References · 14

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  3. 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
  4. 2021SMIL: Multimodal Learning with Severely Missing Modalitycited by 243
  5. 2022Are Multimodal Transformers Robust to Missing Modality?cited by 183
  6. 2023Multi-modal Learning with Missing Modality via Shared-Specific Feature Modellingcited by 159
  7. 2023Multimodal Prompting with Missing Modalities for Visual Recognitioncited by 121
  8. 2022Filter-enhanced MLP is All You Need for Sequential Recommendationcited by 96
  9. 2022Towards Good Practices for Missing Modality Robust Action Recognitioncited by 53
  10. 2023CLIPSyntel: CLIP and LLM Synergy for Multimodal Question Summarization in Healthcarecited by 43
  11. 2024Correlation-Decoupled Knowledge Distillation for Multimodal Sentiment Analysis with Incomplete Modalitiescited by 32
  12. 2020A Survey of Information Cascade Analysis: Models, Predictions, and Recent Advancescited by 26
2 more
  1. 2022Exploiting modality-invariant feature for robust multimodal emotion recognition with missing modalitiescited by 16
  2. 2023Towards Robust Multimodal Prompting With Missing Modalitiescited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 10.

15 papers.

Sort:

By area:

  1. 2025Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generationcited by 42
  2. 2025Synergistic Prompting for Robust Visual Recognition with Missing Modalitiescited by 8
  3. 2025TAMEing Long Contexts in Personalization: Towards Training-Free and State-Aware MLLM Personalized Assistantcited by 6
  4. 2026MM-StanceDet: Retrieval-Augmented Multi-modal Multi-agent Stance Detectioncited by 3
  5. 2025Empowering Multimodal LLMs with External Tools: A Comprehensive Surveycited by 2
  6. 2026RAID: Retrieval-Augmented Anomaly Detectioncited by 2
  7. 2026WISER: Wider Search, Deeper Thinking, and Adaptive Fusion for Training-Free Zero-Shot Composed Image Retrievalcited by 2
  8. 2026OC-Distill: Ontology-aware Contrastive Learning with Cross-Modal Distillation for ICU Risk Predictioncited by 2
  9. 2025Re-coding for Uncertainties: Edge-awareness Semantic Concordance for Resilient Event-RGB Segmentationcited by 1
  10. 2026E2PL: Effective and Efficient Prompt Learning for Incomplete Multi-view Multi-Label Class Incremental Learningcited by 0
  11. 2026Enhancing Foundation VLM Robustness to Missing Modality: Scalable Diffusion for Bi-directional Feature Restorationcited by 0
  12. 2026AOEPT: Breaking the Implicit Modality-Reduction Bottleneck in Modality-Missing Prompt Tuningcited by 0
3 more
  1. 2026C2MOE: Consistency and Complementarity-guided Mixture of Experts for Incomplete Multimodal Emotion Learningcited by 0
  2. 2026I Don't Miss You, but I Do: Self-Explanation Faithfulness of Modality Missingness in Vision-Language Modelscited by 0
  3. 2026PACER: Progressive Availability-Conditioned Evidence Routing for Radiology Report Generation under Incomplete Clinical Contextcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Borrowing Treasures from Neighbors: In-Context Learning for Multimodal Learning with Missing Modalities and Data Scarcity
  2. 2024Reconstruct before Query: Continual Missing Modality Learning with Decomposed Prompt Collaboration
  3. 2023Conditional Prompt Tuning for Multimodal Fusion
  4. 2026Modality Relevance is not Modality Utility: Post-hoc Selective Modality Escalation for Cost-Aware Multimodal RAG
  5. 2026Learning from Reliable Latent Prompts for Visual Recognition with Missing Modalities
  6. 2026GRIP: Feedback-Guided Prompt Retrieval for Large Multimodal Models
  7. 2024ModalPrompt: Towards Efficient Multimodal Continual Instruction Tuning with Dual-Modality Guided Prompt
  8. 2026DeLo: Dual Decomposed Low-Rank Experts Collaboration for Continual Missing Modality Learning