# Synergistic Prompting for Robust Visual Recognition with Missing Modalities

arXiv: https://arxiv.org/abs/2507.07802 · HTML: https://mlsanity.com/paper/2507.07802/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 51 references that are in the map.
- First posted on arXiv 10 Jul 2025.

## Builds on (most cited first, 30 of 31)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [ViViT: A Video Vision Transformer](https://mlsanity.com/paper/2103.15691/) (2021)
- [ShareGPT4V: Improving Large Multi-Modal Models with Better Captions](https://mlsanity.com/paper/2311.12793/) (2023)
- [MaPLe: Multi-modal Prompt Learning](https://mlsanity.com/paper/2210.03117/) (2022)
- [DriveLM: Driving with Graph Visual Question Answering](https://mlsanity.com/paper/2312.14150/) (2023)
- [The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes](https://mlsanity.com/paper/2005.04790/) (2020)
- [Gated Multimodal Units for Information Fusion](https://mlsanity.com/paper/1702.01992/) (2017)
- [CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Models](https://mlsanity.com/paper/2109.11797/) (2021)
- [RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concrete](https://mlsanity.com/paper/2502.21257/) (2025)
- [Cross-Modal Implicit Relation Reasoning and Aligning for Text-to-Image Person Retrieval](https://mlsanity.com/paper/2303.12501/) (2023)
- [Multimodal Prompting with Missing Modalities for Visual Recognition](https://mlsanity.com/paper/2303.03369/) (2023)
- [Deep Multimodal Learning with Missing Modality: A Survey](https://mlsanity.com/paper/2409.07825/) (2024)
- [MixGen: A New Multi-Modal Data Augmentation](https://mlsanity.com/paper/2206.08358/) (2022)
- [DePT: Decoupled Prompt Tuning](https://mlsanity.com/paper/2309.07439/) (2023)
- [Cross-Modal Causal Relational Reasoning for Event-Level Visual Question Answering](https://mlsanity.com/paper/2207.12647/) (2022)
- [Dynamic Adapter Meets Prompt Tuning: Parameter-Efficient Transfer Learning for Point Cloud Analysis](https://mlsanity.com/paper/2403.01439/) (2024)
- [Do You Remember? Dense Video Captioning with Cross-Modal Memory Retrieval](https://mlsanity.com/paper/2404.07610/) (2024)
- [How to Configure Good In-Context Sequence for Visual Question Answering](https://mlsanity.com/paper/2312.01571/) (2023)
- [Cross-Modal Prototype based Multimodal Federated Learning under Severely Missing Modality](https://mlsanity.com/paper/2401.13898/) (2024)
- [MapFusion: A Novel BEV Feature Fusion Network for Multi-modal Map Construction](https://mlsanity.com/paper/2502.04377/) (2025)
- [Missing Modality Prediction for Unpaired Multimodal Learning via Joint Embedding of Unimodal Models](https://mlsanity.com/paper/2407.12616/) (2024)
- [What Foundation Models can Bring for Robot Learning in Manipulation : A Survey](https://mlsanity.com/paper/2404.18201/) (2024)
- [Deep Correlated Prompting for Visual Recognition with Missing Modalities](https://mlsanity.com/paper/2410.06558/) (2024)
- [A Survey on Safe Multi-Modal Learning System](https://mlsanity.com/paper/2402.05355/) (2024)
- [Retrieval-Augmented Dynamic Prompt Tuning for Incomplete Multimodal Learning](https://mlsanity.com/paper/2501.01120/) (2025)

## Built on by (most cited first, 8 of 8)

- [MoRA: Missing Modality Low-Rank Adaptation for Visual Recognition](https://mlsanity.com/paper/2511.06225/) (2025)
- [Rethinking Efficient Mixture-of-Experts for Remote Sensing Modality-Missing Classification](https://mlsanity.com/paper/2511.11460/) (2025)
- [Retrieving to Recover: Towards Incomplete Audio-Visual Question Answering via Semantic-consistent Purification](https://mlsanity.com/paper/2604.10695/) (2026)
- [Enhancing Foundation VLM Robustness to Missing Modality: Scalable Diffusion for Bi-directional Feature Restoration](https://mlsanity.com/paper/2602.03151/) (2026)
- [AOEPT: Breaking the Implicit Modality-Reduction Bottleneck in Modality-Missing Prompt Tuning](https://mlsanity.com/paper/2605.24816/) (2026)
- [Learning from Reliable Latent Prompts for Visual Recognition with Missing Modalities](https://mlsanity.com/paper/2606.30597/) (2026)
- [GAUGE: Granularity-Adaptive Counterfactual Gating of Evidence for Incomplete Multimodal Classification](https://mlsanity.com/paper/2608.05608/) (2026)
- [Test-Time Logit Prompting for Source-Free Missing Modality Adaptation](https://mlsanity.com/paper/2609.02039/) (2026)

## Abstract (arXiv)

Large-scale multi-modal models have demonstrated remarkable performance across various visual recognition tasks by leveraging extensive paired multi-modal training data. However, in real-world applications, the presence of missing or incomplete modality inputs often leads to significant performance degradation. Recent research has focused on prompt-based strategies to tackle this issue; however, existing methods are hindered by two major limitations: (1) static prompts lack the flexibility to adapt to varying missing-data conditions, and (2) basic prompt-tuning methods struggle to ensure reliable performance when critical modalities are missing.To address these challenges, we propose a novel Synergistic Prompting (SyP) framework for robust visual recognition with missing modalities. The proposed SyP introduces two key innovations: (I) a Dynamic Adapter, which computes adaptive scaling factors to dynamically generate prompts, replacing static parameters for flexible multi-modal adaptation, and (II) a Synergistic Prompting Strategy, which combines static and dynamic prompts to balance information across modalities, ensuring robust reasoning even when key modalities are missing. The proposed SyP achieves significant performance improvements over existing approaches across three widely-used visual recognition datasets, demonstrating robustness under diverse missing rates and conditions. Extensive experiments and ablation studies validate its effectiveness in handling missing modalities, highlighting its superior adaptability and reliability.