AOEPT: Breaking the Implicit Modality-Reduction Bottleneck in Modality-Missing Prompt Tuning
Not on the map. Why?
Abstract (arXiv)
Deploying multimodal systems in real-world environments often entails handling modality-missing scenarios, where one or more modalities are unavailable. While recent studies address this challenge for the general Multimodal Transformer (MT) architecture via prompt tuning, we identify a fundamental limitation in these methods: the Implicit Modality-Reduction bottleneck. By conditioning prompts solely on the observed modalities, they inadvertently restrict the reasoning scope of MTs to the modality-reduced subspace, cutting off access to the latent information sources of the missing modalities. To overcome this limitation, we propose AOEPT, which pioneers a novel modal-contextualized prompting fashion. Specifically, we introduce lightweight Modal-Contextualized Prompts (MCPs) that distill global modality-wise priors from training data, serving as latent repositories of the information sources for missing modalities. Conditioned on the remaining modalities, these MCPs are instantiated into instance-aware prompts that selectively augment missing-modality information for each sample, thereby restoring the reasoning scope of MTs beyond the observed-modality-only subspace. Experiments across various multimodal benchmarks and backbones confirm the strong performance of AOEPT, with minimal computational overhead.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 29 of its 46 references that are in the map. First posted on arXiv 24 May 2026.
Builds on
29 of 46 references.
References · 29
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2021Learning to Prompt for Vision-Language Modelscited by 2,269
- 2022Visual Prompt Tuningcited by 1,674
- 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
- 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
- 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 637
- 2020The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memescited by 485
- 2022Multimodal Learning with Transformers: A Surveycited by 399
- 2022Prompt-aligned Gradient for Prompt Tuningcited by 339
- 2023Visual-Language Prompt Tuning with Knowledge-guided Context Optimizationcited by 307
- 2017Gated Multimodal Units for Information Fusioncited by 244
- 2021SMIL: Multimodal Learning with Severely Missing Modalitycited by 243
17 more
- 2022Are Multimodal Transformers Robust to Missing Modality?cited by 183
- 2023Multi-modal Learning with Missing Modality via Shared-Specific Feature Modellingcited by 159
- 2023Multimodal Prompting with Missing Modalities for Visual Recognitioncited by 121
- 2018Words Can Shift: Dynamically Adjusting Word Representations Using Nonverbal Behaviorscited by 89
- 2024Deep Multimodal Learning with Missing Modality: A Surveycited by 89
- 2024Revisiting the Power of Prompt for Visual Tuningcited by 32
- 2025SimMLM: A Simple Framework for Multi-modal Learning with Missing Modalitycited by 22
- 2024Missing Modality Prediction for Unpaired Multimodal Learning via Joint Embedding of Unimodal Modelscited by 20
- 2024Deep Correlated Prompting for Visual Recognition with Missing Modalitiescited by 17
- 2023Towards Robust Multimodal Prompting With Missing Modalitiescited by 16
- 2025Retrieval-Augmented Dynamic Prompt Tuning for Incomplete Multimodal Learningcited by 15
- 2022Prompting through Prototype: A Prototype-based Prompt Learning on Pretrained Vision-Language Modelscited by 12
- 2024Understanding and Mitigating Miscalibration in Prompt Tuning for Vision-Language Modelscited by 9
- 2025Synergistic Prompting for Robust Visual Recognition with Missing Modalitiescited by 8
- 2025Ask and Remember: A Questions-Only Replay Strategy for Continual Visual Question Answeringcited by 7
- 2026Nip Rumors in the Bud: Retrieval-Guided Topic-Level Adaptation for Test-Time Fake News Video Detectioncited by 1
- 2026Shedding the Facades, Connecting the Domains: Detecting Shifting Multimodal Hate Video with Test-Time Adaptationcited by 1
Built on by
Not cited yet.